How is Gaze Influenced by Image Transformations? Dataset and Model

• Published in: IEEE transactions on image processing Vol. 29; pp. 2287 - 2300
• Main Authors: Che, Zhaohui, Borji, Ali, Zhai, Guangtao, Min, Xiongkuo, Guo, Guodong, Le Callet, Patrick
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE); Institute of Electrical and Electronics Engineers
• Subjects: Artificial Intelligence; Computer Science; Data augmentation; Data models; Datasets; Eye movements; generative adversarial networks; Human gaze; Human performance; Image Processing; Image resolution; Mathematical model; model robustness; Modules; Observers; Performance degradation; Robustness; Salience; saliency prediction; Semantics; Transformations; Visualization
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2019.2945857

Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time-consuming and expensive. Most of current studies on human attention and saliency modeling have used high-quality stereotype stimuli. In real world, however, captured images undergo various types of transformations. Can we use these transformations to augment existing saliency datasets? Here, we first create a novel saliency dataset including fixations of 10 observers over 1900 images degraded by 19 types of transformations. Second, by analyzing eye movements, we find that observers look at different locations over transformed versus original images. Third, we utilize the new data over transformed images, called data augmentation transformation (DAT), to train deep saliency models. We find that label-preserving DATs with negligible impact on human gaze boost saliency prediction, whereas some other DATs that severely impact human gaze degrade the performance. These label-preserving valid augmentation transformations provide a solution to enlarge existing saliency datasets. Finally, we introduce a novel saliency model based on generative adversarial networks (dubbed GazeGAN). A modified U-Net is utilized as the generator of the GazeGAN, which combines classic "skip connection" with a novel "center-surround connection" (CSC) module. Our proposed CSC module mitigates trivial artifacts while emphasizing semantic salient regions, and increases model nonlinearity, thus demonstrating better robustness against transformations. Extensive experiments and comparisons indicate that GazeGAN achieves state-of-the-art performance over multiple datasets. We also provide a comprehensive comparison of 22 saliency models on various transformed scenes, which contributes a new robustness benchmark to saliency community. Our code and dataset are available at: https://github.com/CZHQuality/Sal-CFS-GAN.