Deep learning-based image deconstruction method with maintained saliency

• Published in: Neural networks Vol. 155; pp. 224 - 241
• Main Authors: Fujimoto, Keisuke, Hayashi, Kojiro, Katayama, Risa, Lee, Sehyung, Liang, Zhen, Yoshida, Wako, Ishii, Shin
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.11.2022
• Subjects: Attention; Deep Learning; Functional magnetic resonance imaging; Humans; Image transformation; Magnetic Resonance Imaging; Neural Networks, Computer; Saliency map; Variational autoencoder; Deep learning; Variational autoencoder; Functional magnetic resonance imaging; Image transformation; Saliency map; Attention
• ISSN: 0893-6080; 1879-2782
• DOI: 10.1016/j.neunet.2022.08.015

Visual properties that primarily attract bottom-up attention are collectively referred to as saliency. In this study, to understand the neural activity involved in top-down and bottom-up visual attention, we aim to prepare pairs of natural and unnatural images with common saliency. For this purpose, we propose an image transformation method based on deep neural networks that can generate new images while maintaining the consistent feature map, in particular the saliency map. This is an ill-posed problem because the transformation from an image to its corresponding feature map could be many-to-one, and in our particular case, the various images would share the same saliency map. Although stochastic image generation has the potential to solve such ill-posed problems, the most existing methods focus on adding diversity of the overall style/touch information while maintaining the naturalness of the generated images. To this end, we developed a new image transformation method that incorporates higher-dimensional latent variables so that the generated images appear unnatural with less context information but retain a high diversity of local image structures. Although such high-dimensional latent spaces are prone to collapse, we proposed a new regularization based on Kullback–Leibler divergence to avoid collapsing the latent distribution. We also conducted human experiments using our newly prepared natural and corresponding unnatural images to measure overt eye movements and functional magnetic resonance imaging, and found that those images induced distinctive neural activities related to top-down and bottom-up attentional processing.