A Cross-Scale Transformer and Triple-View Attention Based Domain-Rectified Transfer Learning for EEG Classification in RSVP Tasks

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 32; pp. 672 - 683
• Main Authors: Luo, Jie, Cui, Weigang, Xu, Song, Wang, Lina, Chen, Huiling, Li, Yang
• Format: Journal Article
• Language: English
• Published: United States IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Brain; Brain-Computer Interfaces; Brain–computer interface; Classification; Computer applications; Convolution; Deep learning; EEG; Electric Power Supplies; Electroencephalography; Feature extraction; Human-computer interface; Humans; Implants; Information processing; Invariants; Learning; Machine Learning; Representations; RSVP; Source code; Spectrogram; Target detection; Task analysis; Temporal variations; Transfer learning; transformer; Transformers; Visual tasks
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2024.3359191

Rapid serial visual presentation (RSVP)-based brain-computer interface (BCI) is a promising target detection technique by using electroencephalogram (EEG) signals. However, existing deep learning approaches seldom considered dependencies of multi-scale temporal features and discriminative multi-view spectral features simultaneously, which limits the representation learning ability of the model and undermine the EEG classification performance. In addition, recent transfer learning-based methods generally failed to obtain transferable cross-subject invariant representations and commonly ignore the individual-specific information, leading to the poor cross-subject transfer performance. In response to these limitations, we propose a cross-scale Transformer and triple-view attention based domain-rectified transfer learning (CST-TVA-DRTL) for the RSVP classification. Specially, we first develop a cross-scale Transformer (CST) to extract multi-scale temporal features and exploit the dependencies of different scales features. Then, a triple-view attention (TVA) is designed to capture spectral features from triple views of multi-channel time-frequency images. Finally, a domain-rectified transfer learning (DRTL) framework is proposed to simultaneously obtain transferable domain-invariant representations and untransferable domain-specific representations, then utilize domain-specific information to rectify domain-invariant representations to adapt to target data. Experimental results on two public RSVP datasets suggests that our CST-TVA-DRTL outperforms the state-of-the-art methods in the RSVP classification task. The source code of our model is publicly available in https://github.com/ljbuaa/CST_TVA_DRTL .