Multimodal Tweet Sentiment Classification Algorithm Based on Attention Mechanism

• Published in: ECML PKDD 2018 Workshops pp. 68 - 79
• Main Authors: Zou, Peiyu, Yang, Shuangtao
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Communications in Computer and Information Science
• Subjects: Attention mechanism; Multimodal; Sentiment classification
• ISBN: 9783030148799; 3030148793
• ISSN: 1865-0929; 1865-0937
• DOI: 10.1007/978-3-030-14880-5_6

With the rapid development of Internet, multimodal sentiment classification has become an important task in natural language processing research. In this paper, we focus on the sentiment classification of tweets that contains both text and image, a multimodal sentiment classification method for tweets is proposed. In this method Bidirectional-LSTM model is used to extract text modality features and VGG-16 model is used to extract image modality features. Where all features are extracted, a new multimodal feature fusion algorithm based on attention mechanism is used to finish the fusion of text and image features. This fusion method proposed in this paper can give different weights to modalities according to their importance. We evaluated the proposed method on the Chinese Weibo dataset and SentiBank Twitter dataset. The experimental results show method proposed in this paper is better than models that only use single modality feature, and attention based fusion method is more efficient than directly summing or concatenating features from different modalities.