Knowledge Distillation and Contrastive Learning for Robust Multimodal Sentiment Analysis with Missing Data

• Published in: Proceedings (International Conference on Communication Technology. Online) pp. 1856 - 1862
• Main Authors: Wang, Yue, Zhang, Weiliang, Sun, Ning
• Format: Conference Proceeding
• Language: English
• Published: IEEE 18.10.2024
• Subjects: Communications technology; Computer architecture; Contrastive learning; data missing; Data models; Knowledge distillation; Lower bound; multimodal sentiment analysis; Mutual information; Optimization; Robustness; Semantics; Sentiment analysis
• ISSN: 2576-7828
• DOI: 10.1109/ICCT62411.2024.10946552

Multimodal sentiment analysis (MSA) utilizes multimodal such as vison, audio and text, to analyze the inner emotional state of subjects. In the real world, multimodal sentiment data multimodal emotion data are susceptible to disturbances and even complete absence. Performing robust MSA with missing data presents a significant challenge in affective computing. To this end, we propose a novel learning framework called Knowledge distillation and Contrastive learning for Robust MSA (KC4RM), which integrates knowledge distillation and contrastive learning to achieve robust MSA with missing data. KC4RM employs a teacher-student architecture to achieve relation-based knowledge distillation from MSA with complete data to help the model learn complete sentiment semantic information from incomplete multimodal sentiment data. Furthermore, a contrastive learning module is introduced in the student network as a lower bound of mutual information to preserve the key task information from input to the fusion level. The proposed KC4RM is a generative learning framework that can embed various advanced MSA models as the teacher and student networks in the KC4RM to achieve robust MSA with missing data. We conduct extensive experiments to evaluate the proposed method on two MSA benchmarks: MOSI and SIMS. Experimental results show that the KC4RM framework can effectively improve the robustness of MSA in the presence of missing data. Even in the case of severe data missing, the KC4RM framework can still obtain meaningful MSA results. Compared with the current state-of-the-art related methods, the proposed method can obtain superior results.