Learning Spatial-Spectral-Temporal EEG Features With Recurrent 3D Convolutional Neural Networks for Cross-Task Mental Workload Assessment

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 27; no. 1; pp. 31 - 42
• Main Authors: Zhang, Pengbo, Wang, Xue, Zhang, Weihang, Chen, Junfeng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D convolutional neural network; Accident prevention; Adult; Algorithms; Artificial neural networks; Biological neural networks; Brain; Convolutional neural networks; cross-task; Deep learning; EEG; Electroencephalography; Electroencephalography - methods; Feature extraction; Humans; Machine Learning; Male; Mathematics; Mental Processes - physiology; Mental workload; Morlet wavelet; Neural networks; Neural Networks, Computer; recurrent neural network; Space Perception - physiology; Spatial discrimination learning; State of the art; Task analysis; Task complexity; Three-dimensional displays; Topographic mapping; Topographic maps; Wavelet Analysis; Working conditions; Workload; Workloads; Young Adult
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2018.2884641

Mental workload assessment is essential for maintaining human health and preventing accidents. Most research on this issue is limited to a single task. However, cross-task assessment is indispensable for extending a pre-trained model to new workload conditions. Because brain dynamics are complex across different tasks, it is difficult to propose efficient human-designed features based on prior knowledge. Therefore, this paper proposes a concatenated structure of deep recurrent and 3D convolutional neural networks (R3DCNNs) to learn EEG features across different tasks without prior knowledge. First, this paper adds frequency and time dimensions to EEG topographic maps based on a Morlet wavelet transformation. Then, R3DCNN is proposed to simultaneously learn EEG features from the spatial, spectral, and temporal dimensions. The proposed model is validated based on the EEG signals collected from 20 subjects. This paper employs a binary classification of low and high mental workload across spatial n-back and arithmetic tasks. The results show that the R3DCNN achieves an average accuracy of 88.9%, which is a significant increase compared with that of the state-of-the-art methods. In addition, the visualization of the convolutional layers demonstrates that the deep neural network can extract detailed features. These results indicate that R3DCNN is capable of identifying the mental workload levels for cross-task conditions.