EGO-CH: Dataset and fundamental tasks for visitors behavioral understanding using egocentric vision

• Published in: Pattern recognition letters Vol. 131; pp. 150 - 157
• Main Authors: Ragusa, Francesco, Furnari, Antonino, Battiato, Sebastiano, Signorello, Giovanni, Farinella, Giovanni Maria
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.03.2020; Elsevier Science Ltd
• Subjects: Augmented reality; Computer vision; Cultural factors; Cultural heritage; Cultural resources; Datasets; Egocentric vision; First person vision; Historic sites; Learning algorithms; Localization; Machine learning; Object detection; Object recognition; Object retrieval; Pattern recognition; Wearable technology; First person vision; Egocentric vision; Localization; Object retrieval; Object detection
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2019.12.016

•We propose a dataset of egocentric videos for visitor behavior understanding.•Collected by 70 subjects, the data has labels for locations and points of interest.•We propose four tasks for visitor behavior understanding in cultural heritage.•Experiments highlight that the proposed dataset can be a valuable benchmark. Equipping visitors of a cultural site with a wearable device allows to easily collect information about their preferences which can be exploited to improve the fruition of cultural goods with augmented reality. Moreover, egocentric video can be processed using computer vision and machine learning to enable an automated analysis of visitors’ behavior. The inferred information can be used both online to assist the visitor and offline to support the manager of the site. Despite the positive impact such technologies can have in cultural heritage, the topic is currently understudied due to the limited number of public datasets suitable to study the considered problems. To address this issue, in this paper we propose EGOcentric-Cultural Heritage (EGO-CH), the first dataset of egocentric videos for visitors’ behavior understanding in cultural sites. The dataset has been collected in two cultural sites and includes more than 27hours of video acquired by 70 subjects, with labels for 26 environments and over 200 different Points of Interest. A large subset of the dataset, consisting of 60 videos, is associated with surveys filled out by real visitors. To encourage research on the topic, we propose 4 challenging tasks (room-based localization, point of interest/object recognition, object retrieval and survey prediction) useful to understand visitors’ behavior and report baseline results on the dataset.