AB-GRU: An attention-based bidirectional GRU model for multimodal sentiment fusion and analysis

• Published in: Mathematical biosciences and engineering : MBE Vol. 20; no. 10; pp. 18523 - 18544
• Main Authors: Wu, Jun, Zheng, Xinli, Wang, Jiangpeng, Wu, Junwei, Wang, Ji
• Format: Journal Article
• Language: English
• Published: AIMS Press 01.01.2023
• Subjects: attention mechanism; gru; multi-modal fusion; multimedia sentiment analysis
• ISSN: 1551-0018; 1551-0018
• DOI: 10.3934/mbe.2023822

Multimodal sentiment analysis is an important area of artificial intelligence. It integrates multiple modalities such as text, audio, video and image into a compact multimodal representation and obtains sentiment information from them. In this paper, we improve two modules, i.e., feature extraction and feature fusion, to enhance multimodal sentiment analysis and finally propose an attention-based two-layer bidirectional GRU (AB-GRU, gated recurrent unit) multimodal sentiment analysis method. For the feature extraction module, we use a two-layer bidirectional GRU network and connect two layers of attention mechanisms to enhance the extraction of important information. The feature fusion part uses low-rank multimodal fusion, which can reduce the multimodal data dimensionality and improve the computational rate and accuracy. The experimental results demonstrate that the AB-GRU model can achieve 80.9% accuracy on the CMU-MOSI dataset, which exceeds the same model type by at least 2.5%. The AB-GRU model also possesses a strong generalization capability and solid robustness.