Emotional Attention: A Study of Image Sentiment and Visual Attention

• Published in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 7521 - 7531
• Main Authors: Fan, Shaojing, Shen, Zhiqi, Jiang, Ming, Koenig, Bryan L., Xu, Juan, Kankanhalli, Mohan S., Zhao, Qi
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2018
• Subjects: Benchmark testing; Computational modeling; Image annotation; Neural networks; Observers; Semantics; Visualization
• DOI: 10.1109/CVPR.2018.00785

Image sentiment influences visual perception. Emotion-eliciting stimuli such as happy faces and poisonous snakes are generally prioritized in human attention. However, little research has evaluated the interrelationships of image sentiment and visual saliency. In this paper, we present the first study to focus on the relation between emotional properties of an image and visual attention. We first create the EMOtional attention dataset (EMOd). It is a diverse set of emotion-eliciting images, and each image has (1) eye-tracking data collected from 16 subjects, (2) intensive image context labels including object contour, object sentiment, object semantic category, and high-level perceptual attributes such as image aesthetics and elicited emotions. We perform extensive analyses on EMOd to identify how image sentiment relates to human attention. We discover an emotion prioritization effect: for our images, emotion-eliciting content attracts human attention strongly, but such advantage diminishes dramatically after initial fixation. Aiming to model the human emotion prioritization computationally, we design a deep neural network for saliency prediction, which includes a novel subnetwork that learns the spatial and semantic context of the image scene. The proposed network outperforms the state-of-the-art on three benchmark datasets, by effectively capturing the relative importance of human attention within an image. The code, models, and dataset are available online at https://nus-sesame.top/emotionalattention/.