A Transformer based neural network for emotion recognition and visualizations of crucial EEG channels

• Published in: Physica A Vol. 603; p. 127700
• Main Authors: Guo, Jia-Yi, Cai, Qing, An, Jian-Peng, Chen, Pei-Yin, Ma, Chao, Wan, Jun-He, Gao, Zhong-Ke
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.10.2022
• Subjects: Deep learning; EEG; Emotion recognition; Time series analysis; Transformer model; Transformer model; Deep learning; Emotion recognition; Time series analysis; EEG
• ISSN: 0378-4371; 1873-2119
• DOI: 10.1016/j.physa.2022.127700

With the rapid development of artificial intelligence and sensor technology, electroencephalogram-based (EEG) emotion recognition has attracted extensive attention. Various deep neural networks have been applied to it and achieved excellent results in classification accuracy. Except for classification accuracy, the interpretability of the feature extraction process is also considerable for model design for emotion recognition. In this study, we propose a novel neural network model (DCoT) with depthwise convolution and Transformer encoders for EEG-based emotion recognition by exploring the dependence of emotion recognition on each EEG channel and visualizing the captured features. Then we conduct subject-dependent and subject-independent experiments on a benchmark dataset, SEED, which contains EEG data of positive, neutral, and negative emotions. For subject-dependent experiments, the average accuracy of three classification tasks is 93.83%. For subject-independent experiments, the average accuracy of three classification tasks is 83.03%. Additionally, we assess the importance of each EEG channel in emotional activities by the DCoT model and visualize it as brain maps. Furthermore, satisfactory results are obtained by utilizing eight selected crucial EEG channels: FT7, T7, TP7, P3, FC6, FT8, T8, and F8, both in two classification tasks and three classification tasks. Using a small number of EEG channels for emotion recognition can reduce equipment costs and computing costs, which is suitable for practical applications. •We proposed a novel neural network based on Transformer encoders and depthwise convolution for EEG-based emotion recognition.•Our model enhances the capability of extracting and fusing frequency-domain information while maintaining the independence between EEG channels.•The crucial EEG channels captured by the proposed model are visualized as brain maps in this work, and few existing works have done it.•Our model is much more interpretable than existing emotion recognition models by exploring the dependence of emotion recognition on each EEEG channel, which is useful in clinical applications.