Spatio-spectral feature classification combining 3D-convolutional neural networks with long short-term memory for motor movement/imagery

• Published in: Engineering applications of artificial intelligence Vol. 120; p. 105862
• Main Authors: Huang, Wenqie, Chang, Wenwen, Yan, Guanghui, Zhang, Yuchan, Yuan, Yueting
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.04.2023
• Subjects: Brain network analysis; Brain-computer interface (BCI); Convolutional neural networks (CNNs); Electroencephalogram (EEG); Long short-term memory (LSTM); Motor imagery (MI); Motor imagery (MI); Brain-computer interface (BCI); Brain network analysis; Convolutional neural networks (CNNs); Electroencephalogram (EEG); Long short-term memory (LSTM)
• ISSN: 0952-1976; 1873-6769
• DOI: 10.1016/j.engappai.2023.105862

In this paper, we propose a novel EEG classification approach based on the Spatio-spectral feature, aiming to design a motor movement/imagery classification model that extracts multi-domain features with promising performance. Firstly, the difference between motor imagery (MI) tasks and real execution tasks was analyzed by calculating complexity measures of different frequency bands. Then, different connection patterns between MI tasks and real execution tasks were investigated by constructing the PLV and PLI networks. The results of the PLV and PLI brain networks showed that real execution tasks’ network connections intensity was stronger than MI tasks, which meant these two networks could be used to distinguish the EEG signal features of different tasks. Afterward, to fully explain the multi-domain features of EEG signals, we fused the Phase Locking Value (PLV) and Phase-Lag Index (PLI) matrices (spatial-domain) under the subsets of frequency bands (frequency-domain) into a 3-D feature, namely the Spatio-spectral feature. Finally, a 3-D convolutional neural network combined with a long short-term memory (3DCNN-LSTM) was utilized to decode the feature. The results showed that the average accuracy of 10 subjects, 20 subjects, 50 subjects, 80 subjects, and 103 subjects was 85.88%, 83.09%, 76.30%, 75.02%, and 74.54%. Taken together, the proposed method provided promising classification accuracies, superior multi-domain features extraction ability, simpler structure, and robustness to classify the different motor movement/imagery tasks. The results contribute to our understanding of applying the deep learning method to decode EEG multi-domain features in the brain-computer interface (BCI) systems (e.g., MI, emotion recognition, and epileptic seizure classification). •Analyzed the difference between MI tasks and corresponding real execution tasks by calculating complexity measures of different frequency bands. The results showed that the differences between different tasks were more significant in the alpha (α) band in the PLV network and the gamma (γ) band in the PLI network.•Investigated different connection patterns between MI tasks and real execution tasks by constructing the PLV and PLI networks after thresholding. The results showed that real execution tasks' network connections intensity was stronger than MI tasks, which meant these two networks could be used to distinguish EEG signal features of different tasks.•Fused the PLV and PLI matrices (spatial-domain) under the subsets of frequency bands (frequency-domain) into a 3-dimensional feature, namely the Spatio-spectral feature.•Utilized a 3-dimensional convolutional neural network combined with a long short-term memory (3DCNN-LSTM) to decode and classify the Spatio-spectral feature. The average classification accuracy of 10 subjects, 20 subjects, 50 subjects, 80 subjects, and 103 subjects was 85.88%, 83.09%, 76.30%, 75.02%, and 74.54%, respectively.