Improving Cross-State and Cross-Subject Visual ERP-Based BCI With Temporal Modeling and Adversarial Training

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 30; pp. 369 - 379
• Main Authors: Ni, Ziyi, Xu, Jiaming, Wu, Yuwei, Li, Mengfan, Xu, Guizhi, Xu, Bo
• Format: Journal Article
• Language: English
• Published: United States IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation models; adversarial training; Brain; Brain modeling; Brain-computer interface; Brain-Computer Interfaces; Computer applications; cross-state; cross-subject; Deep learning; EEG; Electroencephalography; Electroencephalography - methods; Event-related potentials; Evoked Potentials - physiology; Feature extraction; Human-computer interface; Humans; Implants; Modelling; Muscles; Peripheral nerves; Perturbation; Stochasticity; Task analysis; temporal modeling; Training; Visual tasks; Visualization; Workload; Workloads
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2022.3150007

Brain-computer interface (BCI) is a useful device for people without relying on peripheral nerves and muscles. However, the performance of the event-related potential (ERP)-based BCI declines when applying it to real environments, especially in cross-state and cross-subject conditions. Here we employ temporal modeling and adversarial training to improve the visual ERP-based BCI under different mental workload states and to alleviate the problems above. The rationality of our method is that the ERP-based BCI is based on electroencephalography (EEG) signals recorded from the scalp's surface, continuously changing with time and somewhat stochastic. In this paper, we propose a hierarchical recurrent network to encode all ERP signals in each repetition at the same time and model them with a temporal manner to predict which visual event elicited an ERP. The hierarchical architecture is a simple yet effective method for organizing recurrent layers in a deep structure to model long sequence signals. Taking a cue from recent advances in adversarial training, we further applied dynamic adversarial perturbations to create adversarial examples to enhance the model performance. We conduct our experiments on one published visual ERP-based BCI task with 15 subjects and 3 different auditory workload states. The results indicate that our hierarchical method can effectively model the long sequence EEG raw data, outperform the baselines on most conditions, including cross-state and cross-subject conditions. Finally, we show how deep learning-based methods with limited EEG data can improve ERP-based BCI with adversarial training. Our code is available at https://github.com/aispeech-lab/VisBCI .