Contrastive Knowledge Distillation for Robust Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Sang, Zhongyi, Funakoshi, Kotaro, Okumura, Manabu
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 11.10.2024
Subjects
Computing costs
Cost analysis
Data mining
Human performance
Knowledge management
Missing data
Robustness
Sentiment analysis
Systems analysis
Teachers
Online AccessGet full text

Cover

More Information
Summary:Multimodal sentiment analysis (MSA) systems leverage information from different modalities to predict human sentiment intensities. Incomplete modality is an important issue that may cause a significant performance drop in MSA systems. By generative imputation, i.e., recovering the missing data from available data, systems may achieve robust performance but will lead to high computational costs. This paper introduces a knowledge distillation method, called `Multi-Modal Contrastive Knowledge Distillation' (MM-CKD), to address the issue of incomplete modality in video sentiment analysis with lower computation cost, as a novel non-imputation-based method. We employ Multi-view Supervised Contrastive Learning (MVSC) to transfer knowledge from a teacher model to student models. This approach not only leverages cross-modal knowledge but also introduces cross-sample knowledge with supervision, jointly improving the performance of both teacher and student models through online learning. Our method gives competitive results with significantly lower computational costs than state-of-the-art imputation-based methods.
ISSN:2331-8422