HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument

In this paper, an interactive learning experience is proposed, aiming to involve museum visitors in a personalized experience of the transmittal of cultural knowledge in an active and creative way. The proposed system, called HapticSOUND, consists of three subsystems: (a) the Information, where visi...

Full description

Saved in:
Bibliographic Details
Published inApplied sciences Vol. 13; no. 12; p. 7149
Main Authors Volioti, Christina, Tsagaris, Apostolos, Trigkas, Dimitrios, Iliou, Theodoros, Katsantonis, Menelaos N., Mavridis, Ioannis
Format Journal Article
LanguageEnglish
Published Basel MDPI AG 01.06.2023
Subjects
Algorithms
Cultural heritage
Data mining
Design
Digital music
digital musical instrument
Feedback
Flexibility
Gesture recognition
haptic technology
Keyboards
Learning algorithms
Machine learning
Motion capture
Motor ability
Music industry
Musical instruments
Musical performances
Musicians & conductors
Real time
sensorimotor learning
Sensorimotor system
Sensors
Sound
Subsystems
Usability
User satisfaction
Greece
Online AccessGet full text

Cover

More Information
Summary:In this paper, an interactive learning experience is proposed, aiming to involve museum visitors in a personalized experience of the transmittal of cultural knowledge in an active and creative way. The proposed system, called HapticSOUND, consists of three subsystems: (a) the Information, where visitors are informed about the traditional musical instruments; (b) the Entertainment, where visitors are entertained by playing serious games to virtually assemble traditional musical instruments by a set of 3D objects; and (c) the Interaction, where visitors interact with a digital musical instrument which is an exact 3D-printed replica of a traditional musical instrument, where cameras have been placed to capture user gestures and machine learning algorithms have been implemented for gesture recognition. The museum visitor can interact with the lifelike replica to tactilely and aurally explore the instrument’s abilities, producing sounds guided by the system and receiving real-time visual and audio feedback. Emphasis is given to the Interaction Subsystem, where a pilot study was conducted to evaluate the usability of the subsystem. Preliminary results were promising since the usability was satisfactory, indicating that it is an innovative approach that utilizes sensorimotor learning and machine learning techniques in the context of playing sounds based on real-time gesture and fingering recognition.
ISSN:2076-3417
2076-3417
DOI:10.3390/app13127149