CLUE: Contrastive language-guided learning for referring video object segmentation

• Published in: Pattern recognition letters Vol. 178; pp. 115 - 121
• Main Authors: Gao, Qiqi, Zhong, Wanjun, Li, Jie, Zhao, Tiejun
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.02.2024
• Subjects: Contrastive learning; Deep learning; Multi-modal; Video object segmentation; Video object segmentation; Deep learning; Multi-modal; Contrastive learning
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2023.12.017

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics. •A language-guided contrastive learning and data augmentation method for R-VOS.•A sparse attention method to enhance multi-modal alignment.•An improvement over R-VOS baselines with better identification of textual semantics.