Crowdsourcing authoring of sensory effects on videos

• Published in: Multimedia tools and applications Vol. 78; no. 14; pp. 19201 - 19227
• Main Authors: de Amorim, Marcello Novaes, Saleme, Estêvão Bissoli, Assis Neto, Fábio Ribeiro de, Santos, Celso A. S., Ghinea, Gheorghita
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.07.2019; Springer Nature B.V
• Subjects: Authoring; Computer Communication Networks; Computer Science; Crowdsourcing; Data Structures and Information Theory; Intervals; Metadata; Multimedia; Multimedia Information Systems; Perception; Product safety; Product testing; Response time; Senses; Smell; Special Purpose and Application-Based Systems; User generated content; Visual stimuli; Crowdsourcing; Mulsemedia content; Sensory effects; Multimedia authoring; Multimedia annotation; MPEG-V metadata
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-019-7312-2

Human perception is inherently multi-sensorial involving five traditional senses: sight, hearing, touch, taste, and smell. In contrast to traditional multimedia, based on audio and visual stimuli, mulsemedia seek to stimulate all the human senses. One way to produce multi-sensorial content is authoring videos with sensory effects. These effects are represented as metadata attached to the video content, which are processed and rendered through physical devices into the user’s environment. However, creating sensory effects metadata is not a trivial activity because authors have to identify carefully different details in a scene such as the exact point where each effect starts, finishes, and also its presentation features such as intensity, direction, etc. It is a subjective task that requires accurate human perception and time. In this article, we aim at finding out whether a crowdsourcing approach is suitable for authoring coherent sensory effects associated with video content. Our belief is that the combination of a collective common sense to indicate time intervals of sensory effects with an expert fine-tuning is a viable way to generate sensory effects from the point of view of users. To carry out the experiment, we selected three videos from a public mulsemedia dataset, sent them to the crowd through a cascading microtask approach. The results showed that the crowd can indicate intervals in which users agree that there should be insertions of sensory effects, revealing a way of sharing authoring between the author and the crowd.