Crossmodal Translation Based Meta Weight Adaption for Robust Image-Text Sentiment Analysis

• Published in: IEEE transactions on multimedia pp. 1 - 13
• Main Authors: Zhang, Baozheng, Yuan, Ziqi, Xu, Hua, Gao, Kai
• Format: Journal Article
• Language: English
• Published: IEEE 24.05.2024
• Subjects: Crossmodal Translation; ImageText Sentiment Analysis; Meta Learning; Metalearning; Representation learning; Robustness; Robustness and Reliability; Semantics; Sentiment analysis; Social networking (online); Task analysis
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2024.3405662

Image-Text Sentiment Analysis task has garnered increased attention in recent years due to the surge in user-generated content on social media platforms. Previous research efforts have made noteworthy progress by leveraging the affective concepts shared between vision and text modalities. However, emotional cues may reside exclusively within one of the prevailing modalities, owing to modality independent nature and the potential absence of certain modalities. In this study, we aim to emphasize the significance of modality-independent emotional behaviors, in addition to the modality-invariant behaviors. To achieve this, we propose a novel approach called Crossmodal Translation-Based Meta Weight Adaption (CTMWA). Specifically, our approach involves the construction of the crossmodal translation network, which serves as the encoder. This architecture captures the shared concepts between vision content and text, empowering the model to effectively handle scenarios where either the vision or textual modality is missing. Building upon the translation-based framework, we introduce the strategy of unimodal weight adaption. Leveraging the meta-learning paradigm, our proposed strategy gradually learns to acquire unimodal weights for individual instances from a few hand-crafted meta instances with unimodal annotations. This enables us to modulate the gradients of each modality encoder based on the discrepancy between modalities during model training. Extensive experiments are conducted on three benchmark image-text sentiment analysis datasets, namely MVSA-Single, MVSA-Multiple, and TumEmo. The empirical results demonstrate that our proposed approach achieves the highest performance across all conventional image-text databases. Furthermore, experiments under modality missing settings and case study for reliable sentiment prediction are also conducted further exhibiting superior robustness as well as reliability of the propose approach.