Temporal-constrained parallel graph neural networks for recognizing motion patterns and gait phases in class-imbalanced scenarios

• Published in: Engineering applications of artificial intelligence Vol. 143; p. 110106
• Main Authors: Zhang, Changhe, Yu, Zidong, Wang, Xiaoyun, Chen, Ze-Jian, Deng, Chao
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.03.2025
• Subjects: Class imbalance; Gait phase recognition; Graph neural networks; Hidden markov model; Motion pattern recognition; Multimodal signal fusion; Multimodal signal fusion; Motion pattern recognition; Graph neural networks; Gait phase recognition; Hidden markov model; Class imbalance
• ISSN: 0952-1976
• DOI: 10.1016/j.engappai.2025.110106

Exoskeletons have emerged as a promising technology in the field of motor rehabilitation, particularly for individuals with lower limb motor dysfunction. Related research on human motion intention recognition based on wearable sensors and deep learning is garnering increasing attention from scholars. This study proposes a novel two-stage framework that first performs Motion Pattern Recognition (MPR) and then achieves Gait Phase Recognition (GPR) using surface electromyography and kinematic data from subjects' lower limbs. In view of the short-range dependency of convolutional neural networks, this study exploits Graph Neural Networks (GNNs) to adaptively fuse multimodal signals, and designs several graph generation methods to tackle the issue of lacking original topology in multimodal signals. On this basis, a Parallel GNNs (PGNNs) architecture is constructed to fuse graph-level and domain knowledge features, and a dynamically decaying weighted cross-entropy loss is presented to enhance the recognition performance of PGNNs in class-imbalanced scenarios. Additionally, a categorical hidden Markov model is established using the prior temporal constraints among subject's gait phases to reduce the “phase mutation” errors in PGNNs' predictions without additional training costs. The effectiveness of the proposed framework is verified on a benchmark dataset containing five common lower limb motions of ten healthy subjects. It achieves an average accuracy of 99.97% in the MPR task, and average GPR accuracies of 94.06%, 95.27%, and 95.43% in level walking, upstairs and downstairs motions, respectively, outperforming existing state-of-the-art methods. Experimental results indicate potential support for the controller design of human-robot highly-coupled rehabilitation exoskeletons. •A PGNNs-based two-stage framework is proposed for recognizing lower limb motion patterns and gait phases.•Several graph generation methods are designed for sEMG and kinematic signals.•A DD-WCE loss is proposed to address the class imbalance problem.•A subject-independent CHMM is constructed for gait phase recalibration.