Fusing Frequency-Domain Features and Brain Connectivity Features for Cross-Subject Emotion Recognition

• Published in: IEEE transactions on instrumentation and measurement Vol. 71; pp. 1 - 15
• Main Authors: Chen, Chuangquan, Li, Zhencheng, Wan, Feng, Xu, Leicai, Bezerianos, Anastasios, Wang, Hongtao
• Format: Journal Article
• Language: English
• Published: New York IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Affective computing; Approximate empirical kernel map (AEKM); Brain; brain connectivity (BC) features; Channels; cross-subject emotion recognition; Datasets; Electroencephalography; Emotion recognition; Emotional factors; Emotions; Feature extraction; Frequencies; Frequency domain analysis; frequency-domain (FD) features; fused features; Human-computer interface; Kernel; Motion pictures; Support vector machines
• DOI: 10.1109/TIM.2022.3168927
• ISSN: 0018-9456

Frequency-domain (FD) features reveal the activated patterns of individual local brain regions responding to different emotions, whereas brain connectivity (BC) features involve the coordination of multiple brain regions for generating emotional responses; these two types of features are complementary to each other. To date, the fusion of these two types of features for electroencephalography (EEG)-based cross-subject emotion recognition remains to be fully investigated due to the intersubject variability in EEG signals. In this article, we first attempt to investigate these fused features for cross-subject emotion recognition from multiple perspectives, including critical frequency bands, complementary characteristics for each emotional state, critical channels, and crucial connections, using a fast and robust approximate empirical kernel map-fusion-based support vector machine (AEKM-Fusion-SVM) method. The experimental results on the SJTU emotion EEG dataset (SEED), BCI2020-A, and BCI2020-B datasets reveal that: 1) the AEKM-fusion method improves the effectiveness and efficiency of the fusion of features of different dimensions; 2) the recognition accuracy of the fused features outperforms each individual feature, and this outperformance is more significant in the high-frequency bands (i.e., the beta and gamma bands); 3) the fused features significantly enhance the classification performance for negative emotion; and 4) the fused features built with 27 selected channels achieve comparable performance to that of the fused features built with the full number of channels (i.e., 62 channels), allowing for easier establishment of brain-computer interface (BCI) systems in real-world scenarios. Our study enriches the research of emotion-related brain mechanisms and provides new insight into affective computing.