Learning the Relative Importance of Objects from Tagged Images for Retrieval and Cross-Modal Search

• Published in: International journal of computer vision Vol. 100; no. 2; pp. 134 - 153
• Main Authors: Hwang, Sung Ju, Grauman, Kristen
• Format: Journal Article
• Language: English
• Published: Boston Springer US 01.11.2012; Springer; Springer Nature B.V
• Subjects: Analysis; Artificial Intelligence; Computer Imaging; Computer Science; Correlation analysis; Experiments; Human subjects; Image Processing and Computer Vision; Image retrieval; Information management; Information retrieval; International; Keywords; Learning; Learning strategies; Pattern Recognition; Pattern Recognition and Graphics; Perceptions; Queries; Semantics; Studies; Vision; Vision systems; United Kingdom > UK; Image search; Cross-modal retrieval; Image tags; Image retrieval; Multi-modal retrieval; Kernelized canonical correlation analysis; Object recognition; Auto annotation
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-011-0494-3

We introduce an approach to image retrieval and auto-tagging that leverages the implicit information about object importance conveyed by the list of keyword tags a person supplies for an image. We propose an unsupervised learning procedure based on Kernel Canonical Correlation Analysis that discovers the relationship between how humans tag images (e.g., the order in which words are mentioned) and the relative importance of objects and their layout in the scene. Using this discovered connection, we show how to boost accuracy for novel queries, such that the search results better preserve the aspects a human may find most worth mentioning. We evaluate our approach on three datasets using either keyword tags or natural language descriptions, and quantify results with both ground truth parameters as well as direct tests with human subjects. Our results show clear improvements over approaches that either rely on image features alone, or that use words and image features but ignore the implied importance cues. Overall, our work provides a novel way to incorporate high-level human perception of scenes into visual representations for enhanced image search.