Selective Query-guided Debiasing for Video Corpus Moment Retrieval

• Published in: arXiv.org
• Main Authors: Yoon, Sunjae, Ji Woo Hong, Yoon, Eunseop, Kim, Dahyun, Kim, Junyeong, Yoon, Hee Suk, Yoo, Chang D
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 26.11.2022
• Subjects: Bias; Computer Science - Computer Vision and Pattern Recognition; Object recognition; Qualitative analysis; Queries; Retrieval
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2210.08714

Video moment retrieval (VMR) aims to localize target moments in untrimmed videos pertinent to a given textual query. Existing retrieval systems tend to rely on retrieval bias as a shortcut and thus, fail to sufficiently learn multi-modal interactions between query and video. This retrieval bias stems from learning frequent co-occurrence patterns between query and moments, which spuriously correlate objects (e.g., a pencil) referred in the query with moments (e.g., scene of writing with a pencil) where the objects frequently appear in the video, such that they converge into biased moment predictions. Although recent debiasing methods have focused on removing this retrieval bias, we argue that these biased predictions sometimes should be preserved because there are many queries where biased predictions are rather helpful. To conjugate this retrieval bias, we propose a Selective Query-guided Debiasing network (SQuiDNet), which incorporates the following two main properties: (1) Biased Moment Retrieval that intentionally uncovers the biased moments inherent in objects of the query and (2) Selective Query-guided Debiasing that performs selective debiasing guided by the meaning of the query. Our experimental results on three moment retrieval benchmarks (i.e., TVR, ActivityNet, DiDeMo) show the effectiveness of SQuiDNet and qualitative analysis shows improved interpretability.