Visual complexity analysis using deep intermediate-layer features

• Published in: Computer vision and image understanding Vol. 195; p. 102949
• Main Authors: Saraee, Elham, Jalal, Mona, Betke, Margrit
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.06.2020
• Subjects: Activation energy; Convolutional layers; Convolutional neural network; Deep neural network; Feature extraction; Memorability; Object classification; Scene classification; Visual complexity; Memorability; Visual complexity; Scene classification; Deep neural network; Object classification; Convolutional neural network; Convolutional layers; Feature extraction; Activation energy
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2020.102949

In this paper, we focus on visual complexity, an image attribute that humans can subjectively evaluate based on the level of details in the image. We explore unsupervised information extraction from intermediate convolutional layers of deep neural networks to measure visual complexity. We derive an activation energy metric that combines convolutional layer activations to quantify visual complexity. To show the effectiveness of our proposed metric for various applications, we introduce Savoias, a visual complexity dataset that compromises of more than 1,400 images from seven diverse image categories (e.g., advertisement and interior design). We demonstrate high correlations of our deep neural network-based measure of visual complexity with human-curated ground-truth (GT) scores on various widely used network architectures, e.g., VGG16, ResNet-v2-152, and EfficientNet, and in networks trained on two classification tasks (object and scene classification). This result reveals that intermediate convolutional layers of deep neural networks carry information about the complexity of images that is meaningful to people. Furthermore, we show that our method of measuring visual complexity outperforms traditional methods on Savoias and two other state-of-the-art benchmark datasets. Moreover, we perform extensive analysis on the performance difference between our unsupervised method and a supervised method trained on the feature map, and show that by supervision, we can improve the prediction. Finally, we demonstrate that, within the context of a category, visually more complex images are also more memorable to human observers. [Display omitted] •Unsupervised extraction of information from convolutional layers of deep neural networks.•Unsupervised Activation Energy (UAE) metric to quantify visual complexity.•SAVOIAS, a dataset for the analysis of visual complexity.•High correlation between our UAE method and ground truth.•Within context of category, visually more complex images are more memorable to human.