Electronic music composition thinking using visual learning and visual sensing technology

• Published in: Expert systems
• Main Author: Pi, Jian
• Format: Journal Article
• Language: English
• Published: 22.11.2023
• ISSN: 0266-4720; 1468-0394
• DOI: 10.1111/exsy.13502

Abstract This exploration aims to provide a kind of interactive electronic music composition thinking based on visual learning and visual sensing technology to enhance the thinking process of electronic music composition. Initially, it summarizes the functionalities of the physics components in Unity and analyzes the principles of visual interaction implementation using Virtual Reality (VR) devices and Leap Motion in Unity. Subsequently, the composition of interactive electronic music works is designed and implemented based on ultrasonic sensor technology. Lastly, this exploration focuses on the complete visual design of the audio‐visual integration of Musical Instrument Digital Interface (MIDI) music to represent Cymatics' images of the emotional content of the music. MIDI data can be assigned to provide various mapping possibilities between images and content in music visualization composition. This exploration also designs experiments based on sensory aftereffects and audio‐visual synesthesia to further determine the counterpoint law between the shape of dynamic Cymatics' images and the timbre's audio‐visual synesthetic results. The results indicate that Bass and Kick are distributed in the mid‐low frequency and sub‐low frequency range (20–160 Hz), Vocal and Lead are distributed in the mid‐high frequency range (1280–2560 Hz), and Hihat is distributed in the high‐frequency range (2560–5120 Hz). This exploration utilizes computer technology to create a music visualization method that conforms to the visual expression and aesthetic style of a multi‐sensory experience.