TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning
The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works have dealt with the task by exploring network architectures while learning effective representations has been less explored. In this work, we p...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
21.09.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works have dealt with the task by exploring network architectures while learning effective representations has been less explored. In this work, we propose TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part of the full segments where a person on the screen is actually speaking. This encourages the model to learn effective representations through the natural correspondence of speech and facial movements. Our loss can be jointly optimized with the existing objectives for training ASD models without the need for additional supervision or training data. The experiments demonstrate that our loss can be easily integrated into the existing ASD frameworks, improving their performance. Our method achieves state-of-the-art performances on AVA-ActiveSpeaker and ASW datasets. |
---|---|
AbstractList | The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works have dealt with the task by exploring network architectures while learning effective representations has been less explored. In this work, we propose TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part of the full segments where a person on the screen is actually speaking. This encourages the model to learn effective representations through the natural correspondence of speech and facial movements. Our loss can be jointly optimized with the existing objectives for training ASD models without the need for additional supervision or training data. The experiments demonstrate that our loss can be easily integrated into the existing ASD frameworks, improving their performance. Our method achieves state-of-the-art performances on AVA-ActiveSpeaker and ASW datasets. |
Author | Nam, Kihyun You Jin Kim Jung, Chaeyoung Lee, Suyeon Jang, Youngjoon Rho, Kyeongha Joon Son Chung |
Author_xml | – sequence: 1 givenname: Chaeyoung surname: Jung fullname: Jung, Chaeyoung – sequence: 2 givenname: Suyeon surname: Lee fullname: Lee, Suyeon – sequence: 3 givenname: Kihyun surname: Nam fullname: Nam, Kihyun – sequence: 4 givenname: Kyeongha surname: Rho fullname: Rho, Kyeongha – sequence: 5 fullname: You Jin Kim – sequence: 6 givenname: Youngjoon surname: Jang fullname: Jang, Youngjoon – sequence: 7 fullname: Joon Son Chung |
BookMark | eNqNjMsKwjAURIMoWLX_EHBdaJO-cFdiRUFEsPsS5Kp9mNQkbX_fKH6Aq4GZc2aBpkIKmCCHUBp4aUjIHLla177vkzghUUQddC5425xYvsGHZ6fkUIk7zq6mGgBfOuANKLwFA7aRAo-VeeCP4GUjV4CZFEZx_aWPwJWw9grNbrzV4P5yida7vGB7z76_etCmrGWvhJ1KksZJGMRhQOl_1BtrDUAr |
ContentType | Paper |
Copyright | 2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2023. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection ProQuest Engineering Collection Engineering Database Publicly Available Content (ProQuest) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PTHSS |
ID | FETCH-proquest_journals_28674164133 |
IEDL.DBID | BENPR |
IngestDate | Tue Sep 24 20:14:38 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_28674164133 |
OpenAccessLink | https://www.proquest.com/docview/2867416413/abstract/?pq-origsite=%requestingapplication% |
PQID | 2867416413 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_2867416413 |
PublicationCentury | 2000 |
PublicationDate | 20230921 |
PublicationDateYYYYMMDD | 2023-09-21 |
PublicationDate_xml | – month: 09 year: 2023 text: 20230921 day: 21 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2023 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.4878302 |
SecondaryResourceType | preprint |
Snippet | The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Computer architecture Learning Representations Training |
Title | TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning |
URI | https://www.proquest.com/docview/2867416413/abstract/ |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV3NS8MwFH9sK4I3P_FjjoBeQ2napsWLzNlaBEvRCbuNtE08TGZtK978282L7TwIO4Z8kBfC7339kgdwpXwuHa12qFJCUI-JkOYlC2jBSwc9Cpf_EmRTnrx4Dwt_MYCkfwuDtMoeEw1Ql-8FxshtFnI0HjTm2iLHKEDR2jfVB8X6UZhn7YppDMFijocJW-s2SrOnTbyF8UBbz-4_yDV6JN4DKxOVrPdhINcHsGPol0VzCNlcvK3SWXRNNk4-mRokIs-VFCtZkzvZGtbUmmDolOAEOv0StST4wVQtGjO6-y719Qgu42g-S2i_jWV3aZrln4juMYy09y9PgEgeKhXIXJsKuVeyItRi8EDh24GS-8o5hfG2lc62d5_DLtZPRwIEc8YwautPeaG1bJtPYBjG95PuGHXr8Tv6AQTLh7M |
link.rule.ids | 786,790,12792,21416,33408,33779,43635,43840 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LS8QwEB60RfTmEx-rBvQapGmbFi-yrl2qrqVohb2VtE08rKw1rfj3TWJ2PQh7zoNMCN_MN_mSAbgUIeWecjtYCMZwQFiMq4ZEuKaNpxmFT38FshlNX4OHaTi1CbfOyioXmGiAuvmodY78isRUBw8Kc2_aT6yrRunbVVtCYx3cwFdUxQH3Nsny52WWhdBIxcz-P6A13mO8DW7OWi53YI3Pd2HDiC7rbg_ygr3PslFyjZbUHg0N_qCXlrMZl-iO90YrNUc6YYr0ADz8ZpIj_a2UZJ3pbT9JfduHi3FSjFK8WEZpj0pX_hnmH4CjOD8_BMRpLETEKxUgVEFD6liZQSOhXww0NBTeEQxWzXS8uvkcNtPiaVJO7rPHE9jSFdS1BIJ4A3B6-cVPlZ_tqzO7mT-d7ITP |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=TalkNCE%3A+Improving+Active+Speaker+Detection+with+Talk-Aware+Contrastive+Learning&rft.jtitle=arXiv.org&rft.au=Jung%2C+Chaeyoung&rft.au=Lee%2C+Suyeon&rft.au=Nam%2C+Kihyun&rft.au=Rho%2C+Kyeongha&rft.date=2023-09-21&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |