Affective image recognition with multi-attribute knowledge in deep neural networks

• Published in: Multimedia tools and applications Vol. 83; no. 6; pp. 18353 - 18379
• Main Authors: Zhang, Hao, Luo, Gaifang, Yue, Yingying, He, Kangjian, Xu, Dan
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2024; Springer Nature B.V
• Subjects: Aesthetics; Artificial neural networks; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Design; Emotion recognition; Emotions; Image contrast; Machine learning; Modules; Multimedia; Multimedia Information Systems; Neural networks; Semantics; Special Purpose and Application-Based Systems; Track 6: Computer Vision for Multimedia Applications; Visual details; Affective image recognition; Deep metric learning; Semantics; Multi-attribute
• ISSN: 1573-7721; 1380-7501; 1573-7721
• DOI: 10.1007/s11042-023-16081-7

Incorporating visual attributes such as objects and scene features into deep models has been proved valuable for affective image recognition. In general, the existing works achieve it by either fine-tuning popular CNNs for emotion recognition, or connecting external attributes through additional well-designed modules. However, they do not realize the diversity of emotional representations for different styles of affective images, or utilize the inter-hierarchical correlations in deep models. In this paper, we propose a multi-attribute model which incorporates different visual concepts to solve this problem. The model consists of 2 branch modules from local to global view: one trains a gram encoder to capture local visual details, and the other trains a semantic tokenizer to extract global semantics simultaneously. Through a fusion layer, we represent image sentiments with aggregated attributes. Different from the existing methods, our model is composed of stacked CNNs without additional backbones, and it shows the great ability to learn hierarchical attributes from internal intermediate features. Furthermore, inspired by deep metric learning, we design an emotional contrast loss to consider dynamic polarity embedded in affective images, and optimize the model within cross-entropy loss as well. A comprehensive evaluation on 5 datasets supports that our model outperforms the others.