Deep Reinforcement Learning for Physics-Based Musculoskeletal Simulations of Healthy Subjects and Transfemoral Prostheses' Users During Normal Walking

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 29; pp. 607 - 618
• Main Authors: De Vree, Leanne, Carloni, Raffaella
• Format: Journal Article
• Language: English
• Published: United States IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Actuators; Algorithms; Ankle; Computational modeling; Computer applications; computer simulation; Computing time; Deep learning; Deep Reinforcement Learning (DRL); Gait; Legged locomotion; Machine learning; Muscles; Observational learning; Optimization; Optimization algorithms; Physics; Prostheses; Prosthetics; Reinforcement; Reinforcement learning; Walking
• DOI: 10.1109/TNSRE.2021.3063015
• ISSN: 1534-4320; 1558-0210

This paper proposes to use deep reinforcement learning for the simulation of physics-based musculoskeletal models of both healthy subjects and transfemoral prostheses' users during normal level-ground walking. The deep reinforcement learning algorithm is based on the proximal policy optimization approach in combination with imitation learning to guarantee a natural walking gait while reducing the computational time of the training. Firstly, the optimization algorithm is implemented for the OpenSim model of a healthy subject and validated with experimental data from a public data-set. Afterwards, the optimization algorithm is implemented for the OpenSim model of a generic transfemoral prosthesis' user, which has been obtained by reducing the number of muscles around the knee and ankle joints and, specifically, by keeping only the uniarticular ones. The model of the transfemoral prosthesis' user shows a stable gait, with a forward dynamic comparable to the healthy subject's, yet using higher muscles' forces. Even though the computed muscles' forces could not be directly used as control inputs for muscle-like linear actuators due to their pattern, this study paves the way for using deep reinforcement learning for the design of the control architecture of transfemoral prostheses.