Learning Sound Localization Better From Semantically Similar Samples

The objective of this work is to localize the sound sources in visual scenes. Existing audio-visual works employ contrastive learning by assigning corresponding audio-visual pairs from the same source as positives while randomly mismatched pairs as negatives. However, these negative pairs may contai...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Senocak, Arda, Ryu, Hyeonggon, Kim, Junsik, In So Kweon
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 07.02.2022
Subjects
Audio data
Learning
Sound localization
Sound sources
Online AccessGet full text

Cover

More Information
Summary:The objective of this work is to localize the sound sources in visual scenes. Existing audio-visual works employ contrastive learning by assigning corresponding audio-visual pairs from the same source as positives while randomly mismatched pairs as negatives. However, these negative pairs may contain semantically matched audio-visual information. Thus, these semantically correlated pairs, "hard positives", are mistakenly grouped as negatives. Our key contribution is showing that hard positives can give similar response maps to the corresponding pairs. Our approach incorporates these hard positives by adding their response maps into a contrastive learning objective directly. We demonstrate the effectiveness of our approach on VGG-SS and SoundNet-Flickr test sets, showing favorable performance to the state-of-the-art methods.
ISSN:2331-8422