Wearable Supernumerary Robotic Limb System Using a Hybrid Control Approach Based on Motor Imagery and Object Detection

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 30; pp. 1298 - 1309
• Main Authors: Tang, Zhichuan, Zhang, Lingtao, Chen, Xin, Ying, Jichen, Wang, Xinyang, Wang, Hang
• Format: Journal Article
• Language: English
• Published: United States IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Control systems; EEG; Electroencephalography; Feature extraction; Grasping; Hardware; Hemiplegia; Human engineering; human-robot interaction; Hybrid control; Hybrid systems; Manipulators; Mental task performance; motion imagery; Motion perception; Motor task performance; Object detection; Object recognition; Robots; Subsystems; Supernumerary; Supernumerary robotic limb; Task analysis; upper limb assistance; Wearable technology
• DOI: 10.1109/TNSRE.2022.3172974
• ISSN: 1534-4320; 1558-0210

Motor disorder of upper limbs has seriously affected the daily life of the patients with hemiplegia after stroke. We developed a wearable supernumerary robotic limb (SRL) system using a hybrid control approach based on motor imagery (MI) and object detection for upper-limb motion assistance. SRL system included an SRL hardware subsystem and a hybrid control software subsystem. The system obtained the patient's motion intention through MI electroencephalogram (EEG) recognition method based on graph convolutional network (GCN) and gated recurrent unit network (GRU) to control the left and right movements of SRL, and the object detection technology was used together for a quick grasp of target objects to compensate for the disadvantages when using MI EEG alone like fewer control instructions and lower control efficiency. Offline training experiment was designed to obtain subjects' MI recognition models and evaluate the feasibility of the MI EEG recognition method; online control experiment was designed to verify the effectiveness of our wearable SRL system. The results showed that the proposed MI EEG recognition method (GCN+GRU) could effectively improve the MI classification accuracy (90.04% <inline-formula> <tex-math notation="LaTeX">\pm ~2.36 </tex-math></inline-formula>%) compared with traditional methods; all subjects were able to complete the target object grasping tasks within 23 seconds by controlling the SRL, and the highest average grasping success rate achieved 90.67% in bag grasping task. The SRL system can effectively assist people with upper-limb motor disorder to perform upper-limb tasks in daily life by natural human-robot interaction, and improve their ability of self-help and enhance their confidence of life.