Single-trial stimuli classification from detected P300 for augmented Brain–Computer Interface: A deep learning approach

• Published in: Machine learning with applications Vol. 9; p. 100393
• Main Authors: Leoni, Jessica, Strada, Silvia Carla, Tanelli, Mara, Brusa, Alessandra, Proverbio, Alice Mado
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 15.09.2022; Elsevier
• Subjects: Brain–Computer interface (BCI); Classification; Deep learning; EEG; Deep learning; Brain–Computer interface (BCI); EEG; Classification
• ISSN: 2666-8270; 2666-8270
• DOI: 10.1016/j.mlwa.2022.100393

The purpose of advanced Brain–Computer Interfaces (BCIs) is to connect the human brain with an external device without using the muscular system. To do this, they must effectively process mental activity and infer information on the users’ intentions and directives. This work proposes a novel and explainable BCI system capable of recognizing P300 deflection in single-trial EEGs with higher accuracy compared to the literature gold standard. Moreover, the proposed deep-learning approach allows us to go beyond the mere P300 detection, which is, to our best knowledge, the current state of the art. Indeed, we first identify the P300-related signal in the single-trial EEG signal, and then, we further discriminate the ERPs associated to the detected P300 between visual and auditory stimuli-related. To do this, we employ a CNN–LSTM neural network, which manages a 3D data representation of the acquired EEG signals. The performance of the approach is tested on experiments carried out on 22 subjects, revealing a 82.4% F1-score in P300 identification and 82.4% discriminating between visual and auditory stimuli. The employed algorithmic procedure also reports the most relevant each EEG channels in determining the predictions, adding interpretability to the proposed AI-based tools. These results pave the way for more sophisticated BCIs, capable of extending the set of available actions for the patients. The project was pre-approved by the Research Assessment Committee of the Department of Psychology (CRIP) for minimal risk projects, under the aegis of the Ethical Committee of University of Milano-Bicocca, on May 27th, 2019, protocol number RM-2019-193. •Identify ERP deflection in single trial EEG reduces BCI response time.•Matricial EEG representation simultaneously express spatio-temporal information.•A CNN–LSTM network detect the ERP and map the stimuli to their sensorial domain.•Auditory and visual ERPs are distinguished, increasing BCI potentials.