Leveraging Features from Background and Salient Regions for Automatic Image Annotation

• Published in: Journal of Information Processing Vol. 20; no. 1; pp. 250 - 266
• Main Authors: Fahrmair, Michael, Sarin, Supheakmungkol, Wagner, Matthias, Kameyama, Wataru
• Format: Journal Article
• Language: English
• Published: Information Processing Society of Japan 2012
• ISSN: 1882-6652; 1882-6652
• DOI: 10.2197/ipsjjip.20.250

In this era of information explosion, automating the annotation process of digital images is a crucial step towards efficient and effective management of this increasingly high volume of content. However, this still is a highly challenging task for the research community. One of the main bottlenecks is the lack of integrity and diversity of features. We propose to solve this problem by utilizing 43 image features that cover the holistic content of the image from global to subject, background and scene. In our approach, salient regions and the background are separated without prior knowledge. Each of them together with the whole image are treated independently for feature extraction. Extensive experiments were designed to show the efficiency and the effectiveness of our approach. We chose two publicly available datasets manually annotated with diverse nature of images for our experiments, namely, the Corel5K and ESP Game datasets. We confirm the superior performance of our approach over the use of a single whole image using sign test with p-value < 0.05. Furthermore, our combined feature set gives satisfactory performance compared to recently proposed approaches especially in terms of generalization even with just a simple combination. We also obtain a better performance with the same feature set versus the grid-based approach. More importantly, when using our features with the state-of-the-art technique, our results show higher performance in a variety of standard metrics.