Emotion recognition framework using multiple modalities for an effective human–computer interaction

• Published in: The Journal of supercomputing Vol. 79; no. 8; pp. 9320 - 9349
• Main Authors: Moin, Anam, Aadil, Farhan, Ali, Zeeshan, Kang, Dongwann
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2023; Springer Nature B.V
• Subjects: Arousal; Brain; Compilers; Computer Science; Electroencephalography; Emotion recognition; Emotions; Interpreters; Physiology; Processor Architectures; Programming Languages; Support vector machines; Virtual reality; Multimodal emotion recognition; Electroencephalogram (EEG); Facial features; Machine learning; Human–computer interaction
• ISSN: 0920-8542; 1573-0484
• DOI: 10.1007/s11227-022-05026-w

Human emotions are subjective reactions to objects or events that are related to diverse physiological, behavioral and intellectual changes. The research community is gaining more interest in emotion recognition due to its vast applications including human–computer interaction, virtual reality, self-driving, digital content entertainment, human behavior monitoring, and medicine. Electroencephalogram (EEG) signals that are collected from the brain are playing a massive part in the advancement of brain–computer interface systems. The current techniques that are using EEG signals for emotion recognition are lacking in subject-independent or cross-subject emotion analysis. Additionally, there is a lack of multimodal approaches that combine EEG data with other modalities. In view of the stated deficiencies, this study presents an efficient multimodal strategy for cross-subject emotion recognition utilizing EEG and facial gestures. The proposed method fuses the spectral and statistical features extracted from the EEG data with a histogram of oriented gradients and local binary patterns features extracted from the facial images. Following on, support vector machines, k-nearest neighbor, and ensemble are employed for emotion classification. Additionally, the class misbalance problem is solved using the up-sampling approach. The accuracy of the suggested method is assessed on the dataset of emotion analysis using physiological signals with tenfold cross-validation. The findings of the research study are promising, with the highest accuracy of 97.25% for valence and 96.1% for arousal, respectively.