Perceptive vs reflective: spectator interpretation of multimodal artworks

• Published in: Personal and ubiquitous computing Vol. 27; no. 5; pp. 1965 - 1975
• Main Authors: Bramah, Oliver, Adams, Sue, Morreale, Fabio
• Format: Journal Article
• Language: English
• Published: London Springer London 01.10.2023; Springer Nature B.V
• Subjects: Abstract art; Computer Science; Empirical analysis; Mobile Computing; Music; Musical interpretation; Original Article; Perceptions; Personal Computing; Senses; Sound; Ubiquitous computing; User Interfaces and Human Computer Interaction; Visual music; Sonic experiences; Qualitative; Synaesthesia; Empirical study
• ISSN: 1617-4909; 1617-4917
• DOI: 10.1007/s00779-023-01744-x

This article presents an empirical investigation into spectator interpretation of multimodal artworks. Specifically, this article explores perception of the artwork in two different sensorial modalities — sight and sound — and the effect of their interaction. We selected four abstract paintings and created four acousmatic musical pieces that were composed to reflect the content of the artwork. We then ran a between-subjects experimental study in three conditions: visual only, music only, and a combination of both. A total of 48 participants completed an online survey in which they were asked to report their interpretations of the shown artworks. Following a thematic analysis on the collected data, we clustered participants’ interpretation into two main categories: reflective and perceptive . The combination increased spectators’ attention to the artworks, affected the temporality of the artworks, and created richer understandings of the multimodal works. The study provides knowledge for the inclusion of multimodal experiences in the presentation of art expanding the possibilities for inter-sensory dialogue in the arts.