Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors

• Published in: Scientific reports Vol. 8; no. 1; pp. 13657 - 15
• Main Authors: Marín-Morales, Javier, Higuera-Trujillo, Juan Luis, Greco, Alberto, Guixeres, Jaime, Llinares, Carmen, Scilingo, Enzo Pasquale, Alcañiz, Mariano, Valenza, Gaetano
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 12.09.2018; Nature Publishing Group
• Subjects: 631/114/2397; 639/166/985; 639/705/1042; 704/844/1759; Academic disciplines; Affect (Psychology); Algorithms; Arousal; Brain research; Computer applications; EEG; EKG; Electroencephalography; Emotional behavior; Emotions; Humanities and Social Sciences; Laboratories; Machine learning; multidisciplinary; Nervous system; Nonlinear systems; Physiology; Science; Science (multidisciplinary); Sensors; Virtual reality; Wearable computers; Emotion Recognition System; Total Average Accuracy; Valence Perception; Immersive Virtual Environments (IVE); International Affective Picture System (IAPS)
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-018-32063-4

Affective Computing has emerged as an important field of study that aims to develop systems that can automatically recognize emotions. Up to the present, elicitation has been carried out with non-immersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for affective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were designed to elicit four possible arousal-valence combinations, as described in each quadrant of the Circumplex Model of Affects. An experiment involving the recording of the electroencephalography (EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features was extracted from these signals using various state-of-the-art metrics that quantify brain and cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classifier to predict the subject’s arousal and valence perception. The model’s accuracy was 75.00% along the arousal dimension and 71.21% along the valence dimension. Our findings validate the use of Immersive Virtual Environments to elicit and automatically recognize different emotional states from neural and cardiac dynamics; this development could have novel applications in fields as diverse as Architecture, Health, Education and Videogames.