Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 28; no. 5; pp. 1226 - 1238
• Main Authors: Jeong, Ji-Hoon, Shim, Kyung-Hwan, Kim, Dong-Joo, Lee, Seong-Whan
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.05.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Brain; Brain–machine interface (BMI); Convolution; Correlation analysis; Correlation coefficient; Correlation coefficients; Decoding; Deep learning; EEG; electroencephalogram (EEG); Electroencephalography; Extremities; Image acquisition; Interfaces; intuitive robotic arm control; Long short-term memory; Machine learning; Man-machine interfaces; Manipulators; motor imagery; Neural networks; Robot arms; Robot control; Robotics; Task analysis; Three-dimensional displays; Velocity distribution
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2020.2981659

Brain-machine interfaces (BMIs) can be used to decode brain activity into commands to control external devices. This paper presents the decoding of intuitive upper extremity imagery for multi-directional arm reaching tasks in three-dimensional (3D) environments. We designed and implemented an experimental environment in which electroencephalogram (EEG) signals can be acquired for movement execution and imagery. Fifteen subjects participated in our experiments. We proposed a multi-directional convolution neural network-bidirectional long short-term memory network (MDCBN)-based deep learning framework. The decoding performances for six directions in 3D space were measured by the correlation coefficient (CC) and the normalized root mean square error (NRMSE) between predicted and baseline velocity profiles. The grand-averaged CCs of multi-direction were 0.47 and 0.45 for the execution and imagery sessions, respectively, across all subjects. The NRMSE values were below 0.2 for both sessions. Furthermore, in this study, the proposed MDCBN was evaluated by two online experiments for real-time robotic arm control, and the grand-averaged success rates were approximately 0.60 (±0.14) and 0.43 (±0.09), respectively. Hence, we demonstrate the feasibility of intuitive robotic arm control based on EEG signals for real-world environments.