Top-down saliency detection driven by visual classification

• Published in: Computer vision and image understanding Vol. 172; pp. 67 - 76
• Main Authors: Murabito, Francesca, Spampinato, Concetto, Palazzo, Simone, Giordano, Daniela, Pogorelov, Konstantin, Riegler, Michael
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.07.2018
• Subjects: Fully convolutional neural networks; Image classification; Visual attention; 68T45; 68T10; 68T05; Fully convolutional neural networks; 65D05; Visual attention; Image classification
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2018.03.005

•A two-layer deep architecture for top-down saliency detection and fine-grained visual classification.•Driving a saliency detector with a specific goal results in better performance than fine-tuning pre-trained saliency models.•Providing explicitly saliency maps to visual classifiers increased classification performance. This paper presents an approach for saliency detection able to emulate the integration of the top-down (task-controlled) and bottom-up (sensory information) processes involved in human visual attention. In particular, we first learn how to generate saliency when a specific visual task has to be accomplished. Afterwards, we investigate if and to what extent the learned saliency maps can support visual classification in nontrivial cases. To achieve this, we propose SalClassNet, a CNN framework consisting of two networks jointly trained: a) the first one computing top-down saliency maps from input images, and b) the second one exploiting the computed saliency maps for visual classification. To test our approach, we collected a dataset of eye-gaze maps, using a Tobii T60 eye tracker, by asking several subjects to look at images from the Stanford Dogs dataset, with the objective of distinguishing dog breeds. Performance analysis on our dataset and other saliency benchmarking datasets, such as POET, showed that SalClassNet outperforms state-of-the-art saliency detectors, such as SalNet and SALICON. Finally, we also analyzed the performance of SalClassNet in a fine-grained recognition task and found out that it yields enhanced classification accuracy compared to Inception and VGG-19 classifiers. The achieved results, thus, demonstrate that 1) conditioning saliency detectors with object classes reaches state-of-the-art performance, and 2) explicitly providing top-down saliency maps to visual classifiers enhances accuracy.