Exploration of deep learning-driven multimodal information fusion frameworks and their application in lower limb motion recognition

• Published in: Biomedical signal processing and control Vol. 96; p. 106551
• Main Authors: Zhang, Changhe, Yu, Zidong, Wang, Xiaoyun, Chen, Ze-Jian, Deng, Chao, Xie, Sheng Quan
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.10.2024
• Subjects: Deep learning; Inter-subject prediction; Lower limb motion recognition; Multimodal information fusion; Transfer learning; Deep learning; Inter-subject prediction; Lower limb motion recognition; Multimodal information fusion; Transfer learning
• ISSN: 1746-8094
• DOI: 10.1016/j.bspc.2024.106551

•Four end-to-end LLMR frameworks are presented, including CNN-RNNs, CNN-GNNs and two types of CNNs.•Existing mainstream network architectures are compared and evaluated from multiple perspectives.•A novel master nodes-based graph generation method is proposed in the CNN-GNNs framework.•Validation of effectiveness is conducted in both intra-subject and inter-subject scenarios.•The proposed frameworks perform well in distinct tasks for both healthy subjects and patients with stroke. Research on Lower Limb Motion Recognition (LLMR) based on various wearable sensors has been widely applied in exoskeleton robots, exercise rehabilitation, etc. Typically, employing multimodal information tends to yield higher accuracy and stronger robustness compared to using unimodal information. Due to the inevitable reliance on feature engineering in shallow machine learning-based LLMR methods, this study leverages the powerful non-linear feature mapping capability of deep learning (DL) to construct several end-to-end LLMR frameworks, including: Convolutional Neural Networks (CNNs), CNN-Recurrent Neural Networks (RNNs) and CNN-Graph Neural Networks (GNNs). The effectiveness of the proposed frameworks is verified in distinct tasks, including the recognition of seven types of lower limb motions in healthy subjects and three types of motions in patients with stroke, as well as the phase recognition task during the sit-to-stand (SitTS) process in patients with stroke, achieving the highest mean accuracy of 95.198 %, 99.784 %, and 99.845 %, respectively. Further research and integration of two transfer learning techniques, adaptive Batch Normalization (BN) and model fine-tuning, significantly enhance the applicability of the proposed frameworks in inter-subject prediction. Additionally, systematic analyses are conducted to assess the strengths and weaknesses of different models in terms of recognition performance, complexity, and adaptability to variations in the number of modalities and sensor channels. Experimental results indicate that the proposed frameworks hold promise in providing potential support for the development of human-robot collaborative lower limb exoskeletons or rehabilitation robots.