Context-aware network with foreground recalibration for grounding natural language in video

• Published in: Neural computing & applications Vol. 33; no. 16; pp. 10485 - 10502
• Main Authors: Chen, Cheng, Gu, Xiaodong
• Format: Journal Article
• Language: English
• Published: London Springer London 01.08.2021; Springer Nature B.V
• Subjects: Ablation; Artificial Intelligence; Background noise; Computational Biology/Bioinformatics; Computational Science and Engineering; Computer Science; Context; Data Mining and Knowledge Discovery; Image Processing and Computer Vision; Matching; Modules; Natural language; Original Article; Post-processing; Probability and Statistics in Computer Science; Queries; Query languages; Segments; Semantics; Text-based moment retrieval; Semantic modulation based on residual module; Grounding natural language in video; Foreground recalibration
• ISSN: 0941-0643; 1433-3058
• DOI: 10.1007/s00521-021-05807-z

Grounding natural language in video aims at retrieving a matching moment in a long, untrimmed video described by a referring natural language query. It is a challenging issue due to the dominating influence from noise background in untrimmed video and the complex temporal relationships introduced by the query. Existing methods treat different candidate segments separately in a matching and aligning manner and thus neglect that different target segments require different levels of context information. In this paper, we present the semantic modulation residual module, a novel single-shot feed-forward residual network that explicitly integrates various temporal scale features and introduces less noise to the final moments representation with the guide of query semantic information. To establish more fine-grained interactions between different moments, a global interaction module is embedded in the network. Moreover, the data imbalance issue caused by the sparse annotated moments weakens the effect of binary cross-entropy criterion. Therefore, we design a foreground recalibration mechanism to enhance the intra-class consistency and highlight the positive moments. We evaluate our method on three benchmark datasets i.e., TACoS, Charades-STA and ActivityNet Captions, achieving state-of-the-art performance without any post-processing. In particular, we reach 32.17%, 45.11% and 43.76% under the metric Rank@1, IoU@0.5 on TACoS, Charades-STA and ActivityNet Captions, respectively. Furthermore, ablation studies were performed to show the effectiveness of individual components in our proposed method. We hope that the proposed method can serve as a strong and simple alternative for fine-grained video retrieval.