Emotion Recognition of Subjects with Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 31; p. 1
• Main Authors: Li, Dahua, Liu, Jiayin, Yang, Yi, Hou, Fazheng, Song, Haotian, Song, Yu, Gao, Qiang, Mao, Zemin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Brain; Classification; Clips; Deep learning; EEG; Electroencephalogram topographic map; Electroencephalography; Electroencephalography - methods; Emotion recognition; Emotional factors; Emotions; Emotions - physiology; Entropy; Facial Expression; Feature extraction; Geologic depressions; Hearing; Hearing Loss; Humans; Impairment; Individuals with hearing impairment; Machine learning; Nervous system; Occipital lobes; Rehabilitation; Topographic mapping; Topographic maps; Topography
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2022.3225948

Emotion analysis has been employed in many fields such as human-computer interaction, rehabilitation, and neuroscience. But most emotion analysis methods mainly focus on healthy controls or depression patients. This paper aims to classify the emotional expressions in individuals with hearing impairment based on EEG signals and facial expressions. Two kinds of signals were collected simultaneously when the subjects watched affective video clips, and we labeled the video clips with discrete emotional states (fear, happiness, calmness, and sadness). We extracted the differential entropy (DE) features based on EEG signals and converted DE features into EEG topographic maps (ETM). Next, the ETM and facial expressions were fused by the multichannel fusion method. Finally, a deep learning classifier CBAM_ResNet34 combined Residual Network (ResNet) and Convolutional Block Attention Module (CBAM) was used for subject-dependent emotion classification. The results show that the average classification accuracy of four emotions recognition after multimodal fusion achieves 78.32%, which is higher than 67.90% for facial expressions and 69.43% for EEG signals. Moreover, visualization by the Gradient-weighted Class Activation Mapping (Grad-CAM) of ETM showed that the prefrontal, temporal and occipital lobes were the brain regions closely related to emotional changes in individuals with hearing impairment.