Cross Modal Sentiment Analysis of Image Text Fusion Based on Bi LSTM and B-CNN

• Published in: Informatica (Ljubljana) Vol. 48; no. 21; pp. 95 - 111
• Main Authors: Fang, Yuan, Wang, Yi
• Format: Journal Article
• Language: English
• Published: Ljubljana Slovenian Society Informatika / Slovensko drustvo Informatika 01.12.2024
• Subjects: Accuracy; Algorithms; Artificial neural networks; Classification; Data analysis; Deep learning; Emotions; Experiments; Internet; Methods; Neural networks; Recognition; Sentiment analysis; China
• ISSN: 0350-5596; 1854-3871
• DOI: 10.31449/inf.v48i21.6767

Due to the different modalities of data such as images and text, the difficulty of sentiment analysis increases. To achieve cross-modal sentiment analysis, the study firstly designs a cross-modal sentiment analysis method based on bi-directional long and short-term memory networks and bi-linear convolutional neural networks. At the same time, concepts such as image attributes are introduced in the experiment to detect irony in graphic and textual data. Finally, a hybrid strategy cross-modal sentiment analysis method is established in the experiment. After comparison, the proposed method has the highest subject working characteristic curve and PR, which are 5% and 3% higher than the comparative methods, respectively. The model has the lowest error take, with a minimum value of only 0.71%. The average F1 value and average accuracy reached 92.61% and 88.97%, respectively. When the validation set size is 400, the recognition time of the proposed method is 2.1 seconds. When iterating 50, the recognition time of this method is 0.9 seconds. In practical applications, the proposed method has accurately analyzed six types of graphic and textual content with different emotional tendencies. This method has the best detection results for both single graphic and cross-modal modes.