Temporal shift residual network for EEG-based emotion recognition: A 3D feature image sequence approach

• Published in: Multimedia tools and applications Vol. 83; no. 15; pp. 45739 - 45759
• Main Authors: Chen, Yu, Zhang, Haopeng, Long, Jun, Xie, Yining
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2024; Springer Nature B.V
• Subjects: Accuracy; Arousal; Computer Communication Networks; Computer Science; Data exchange; Data Structures and Information Theory; Electroencephalography; Emotion recognition; Emotions; Feature maps; Image sequencing; Modelling; Modules; Multimedia Information Systems; Representations; Special Purpose and Application-Based Systems; Track 2: Medical Applications of Multimedia; emotion recognition; EEG; robust representation; DEAP
• ISSN: 1573-7721; 1380-7501; 1573-7721
• DOI: 10.1007/s11042-023-17142-7

In EEG-based emotion recognition, finding EEG representations that maintain both temporal and spatial features is crucial. This study aims to identify robust representations from EEG independent of subject differences and discriminative. We convert EEG data into feature image sequences with 3D representation, which fully preserve the spatial, spectral and temporal structure of the EEG signal. However, existing models ignore the complementarity between spatial-spectral-temporal features, which limits the classification ability of the models to some extent. Therefore, this paper proposes the Temporal Shift Residual Network(TSM-ResNet) based on feature image sequences for EEG emotion recognition. The Temporal Shift Module(TSM), a highly efficient and high-performance temporal modeling module, is utilized. It shifts certain channels of the feature map along the time dimension, facilitating information exchange between adjacent frames. In summary, the integration of feature image sequences, encompassing multi-domain information, and the powerful temporal modeling of TSM-ResNet enable the unified integration of spatial and spectral features while adequately considering temporal sequence features, all without increasing computational costs. The effectiveness of the proposed method is validated on the internationally recognized DEAP dataset, utilizing evaluation metrics such as accuracy, F1 score, and confusion matrix. The results from subject-dependent experiments (ten-fold cross-validation) demonstrate TSM-ResNet's average accuracy of 93.43% for valence and 93.26% for arousal. Additionally, excellent performance is achieved in subject-independent experiments (leave-one-subject-out cross-validation), with accuracy rates of 64.91% for valence and 62.52% for arousal. These findings highlight the advantages of the proposed method in cross-subject and within-subject emotion recognition.