Deep learning with convolutional neural networks for EEG decoding and visualization

• Published in: Human brain mapping Vol. 38; no. 11; pp. 5391 - 5420
• Main Authors: Schirrmeister, Robin Tibor, Springenberg, Jost Tobias, Fiederer, Lukas Dominique Josef, Glasstetter, Martin, Eggensperger, Katharina, Tangermann, Michael, Hutter, Frank, Burgard, Wolfram, Ball, Tonio
• Format: Journal Article
• Language: English
• Published: United States John Wiley & Sons, Inc 01.11.2017; John Wiley and Sons Inc
• Subjects: Artificial neural networks; Brain; Brain - physiology; Brain mapping; Brain Mapping - methods; Brain-Computer Interfaces; brain–computer interface; brain–machine interface; Computer vision; Decoding; Deep learning; Design; EEG; EEG analysis; electroencephalography; Electroencephalography - methods; end‐to‐end learning; Frequencies; Humans; Imagination - physiology; Language; Learning algorithms; Machine Learning; Mapping; model interpretability; Motor Activity - physiology; Neural networks; Neural Pathways - physiology; Space Perception - physiology; Visualization; brain mapping; electroencephalography; end-to-end learning; brain-computer interface; EEG analysis; model interpretability; machine learning; brain-machine interface
• ISSN: 1065-9471; 1097-0193; 1097-0193
• DOI: 10.1002/hbm.23730

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end‐to‐end learning, that is, learning from the raw data. There is increasing interest in using deep ConvNets for end‐to‐end EEG analysis, but a better understanding of how to design and train ConvNets for end‐to‐end EEG decoding and how to visualize the informative EEG features the ConvNets learn is still needed. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed tasks from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching at least as good performance as the widely used filter bank common spatial patterns (FBCSP) algorithm (mean decoding accuracies 82.1% FBCSP, 84.0% deep ConvNets). While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta, and high gamma frequencies, and proved useful for spatially mapping the learned features by revealing the topography of the causal contributions of features in different frequency bands to the decoding decision. Our study thus shows how to design and train ConvNets to decode task‐related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG‐based brain mapping. Hum Brain Mapp 38:5391–5420, 2017. © 2017 Wiley Periodicals, Inc.