CNN-Transformer network for student learning effect prediction using EEG signals based on spatio-temporal feature fusion

• Published in: Applied soft computing Vol. 170; p. 112631
• Main Authors: Xie, Hui, Dong, Zexiao, Yang, Huiting, Luo, Yanxia, Ren, Shenghan, Zhang, Pengyuan, He, Jiangshan, Jia, Chunli, Yang, Yuqiang, Jiang, Mingzhe, Gao, Xinbo, Chen, Xueli
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.02.2025
• Subjects: EEG; Effectiveness of online learning; Free learning sessions; Hybrid CNN-Transformer (HCT) model; Free learning sessions; Hybrid CNN-Transformer (HCT) model; Effectiveness of online learning; EEG
• ISSN: 1568-4946
• DOI: 10.1016/j.asoc.2024.112631

Online education has become one of the most important forms of modern education. The uncertainty of the online learning process and the instability of its outcomes make accurately evaluating the outcomes of online learning a significant challenge. Brain–computer interface (BCI) based on deep learning enables analysis and recognition of complex neural patterns, which is a potential solution to predicting learning outcomes from the corresponding learning process. To validate this feasibility, we propose a hybrid CNN-Transformer (HCT) model to distinguish excellent and average learning outcomes from electroencephalogram (EEG) signals recorded during learning. The CNN part learns local spatial representations of multi-channel EEG, while the sparse self-attention fusion with Transformers learns global inter-correlations especially in long-term temporal dependencies. After model training, Grad-CAM is incorporated into the model to visualize and statistically analyze the features learned by the deep network, providing possible explanations for the model’s decision-making process and the neural patterns during learning. The proposed HCT-learn reached 90.13% accuracy with leave-one-subject-out cross-validation in binary learning outcome classification. It outperforms classical and hybrid solutions like EEGNet and ConFormer by at least 2.5% in accuracy, indicating its efficiency in learning spatial–temporal representations of EEG. This study effectively utilizes the potential of combining deep learning with EEG and lays the foundation for applications that dynamically monitor and predict learning outcomes. It guides teachers to make timely adjustments to their teaching strategies during the teaching process and for students to make timely adjustments to their learning methods during the learning process. [Display omitted] •We propose an experiment protocol that simulates online learning in real study life.•HCT-learn can predict learning performance from the EEG in the learning process.•Fusing spatial and temporal whole-brain EEG reached 90% LOSO accuracy within-dataset.•Grad-CAM visualizations reveal spatial-spectral neural patterns during learning.