User Preferences Modeling and Learning for Pleasing Photo Collage Generation

• Published in: ACM transactions on multimedia computing communications and applications Vol. 12; no. 1; pp. 1 - 23
• Main Authors: Bianco, Simone, Ciocca, Gianluigi
• Format: Journal Article
• Language: English
• Published: 01.08.2015
• Subjects: Computation; Criteria; Learning; Multimedia; Optimization; Placing; State of the art; Tasks
• ISSN: 1551-6857; 1551-6865
• DOI: 10.1145/2801126

In this article, we consider how to automatically create pleasing photo collages created by placing a set of images on a limited canvas area. The task is formulated as an optimization problem. Differently from existing state-of-the-art approaches, we here exploit subjective experiments to model and learn pleasantness from user preferences. To this end, we design an experimental framework for the identification of the criteria that need to be taken into account to generate a pleasing photo collage. Five different thematic photo datasets are used to create collages using state-of-the-art criteria. A first subjective experiment where several subjects evaluated the collages, emphasizes that different criteria are involved in the subjective definition of pleasantness. We then identify new global and local criteria and design algorithms to quantify them. The relative importance of these criteria are automatically learned by exploiting the user preferences, and new collages are generated. To validate our framework, we performed several psycho-visual experiments involving different users. The results shows that the proposed framework allows to learn a novel computational model which effectively encodes an inter-user definition of pleasantness. The learned definition of pleasantness generalizes well to new photo datasets of different themes and sizes not used in the learning. Moreover, compared with two state-of-the-art approaches, the collages created using our framework are preferred by the majority of the users.