Exploiting emotional concepts for image emotion recognition

• Published in: The Visual computer Vol. 39; no. 5; pp. 2177 - 2190
• Main Authors: Yang, Hansen, Fan, Yangyu, Lv, Guoyun, Liu, Shiya, Guo, Zhe
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.05.2023; Springer Nature B.V
• Subjects: Ambiguity; Artificial Intelligence; Classification; Cognition; Computer Graphics; Computer Science; Computer vision; Emotion recognition; Emotions; Image Processing and Computer Vision; Image retrieval; Knowledge; Knowledge representation; Learning; Neural networks; Original Article; Semantics; Sentiment analysis; Visual-semantic embedding; Image emotion recognition; Emotion perception; Knowledge graph
• ISSN: 0178-2789; 1432-2315
• DOI: 10.1007/s00371-022-02472-8

With the increasing number of users express their emotions via images on social media, image emotion recognition attracts much attention of researchers. Different from conventional computer vision tasks, image emotion recognition is inherently more challenging for the ambiguity and subjectivity of emotion. Existing methods are limited to learn a direct mapping from image feature to emotion. However, emotion cognition mechanism in psychology demonstrates that human beings perceive emotion in a stepwise way. Therefore, we propose a novel image emotion recognition method that leverages emotional concepts as intermediary to bridge image and emotion. Specifically, we organize the relationship between concept and emotion in the form of knowledge graph. The relation between image and emotion is explored in the semantic embedding space where the knowledge is encoded into. Then, based on the hierarchical relation of emotions, we propose a multi-task learning deep model to recognize image emotion from visual perspective. Finally, a fusion strategy is proposed to merge the results of both visual-semantic stream and visual stream. Extensive experimental results show that our method outperforms state-of-the-art methods on two public image emotion datasets.