Marker Data Enhancement for Markerless Motion Capture

• Published in: IEEE transactions on biomedical engineering Vol. 72; no. 6; pp. 2013 - 2022
• Main Authors: Falisse, Antoine, Uhlrich, Scott D., Chaudhari, Akshay S., Hicks, Jennifer L., Delp, Scott L.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2025; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Adult; Algorithms; Benchmarks; Biomechanical Phenomena - physiology; Biomedical engineering; Computational modeling; Data models; Deep Learning; Female; Hip; Human motion; Humans; Image Processing, Computer-Assisted - methods; Kinematics; Long short term memory; Male; markerless motion capture; Motion Capture; Movement - physiology; musculoskeletal modeling and simulation; Pose estimation; Posture - physiology; Predictive models; Solid modeling; Synthetic data; Three-dimensional displays; Training; trajectory optimization; Video; Video Recording - methods
• ISSN: 0018-9294; 1558-2531; 1558-2531
• DOI: 10.1109/TBME.2025.3530848

Objective: Human pose estimation models can measure movement from videos at a large scale and low cost; however, open-source pose estimation models typically detect only sparse keypoints, which leads to inaccurate joint kinematics. OpenCap, a freely available service for researchers to measure movement from videos, mitigates this issue using a deep learning model-the marker enhancer-that transforms sparse video keypoints into dense anatomical markers. However, OpenCap performs poorly on movements not included in the training data. Here, we create a much larger and more diverse training dataset and develop a more accurate and generalizable marker enhancer. Methods: We compiled marker-based motion capture data from 1176 subjects and synthesized 1433 hours of video keypoints and anatomical markers to train the marker enhancer. We evaluated its accuracy in computing kinematics using both benchmark movement videos and synthetic data representing unseen, diverse movements. Results: The marker enhancer improved kinematic accuracy on benchmark movements (mean error: 4.1<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>, max: 8.7<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>) compared to using video keypoints (mean: 9.6<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>, max: 43.1<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>) and OpenCap's original enhancer (mean: 5.3<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>, max: 11.5<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>). It also better generalized to unseen, diverse movements (mean: 4.1<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>, max: 6.7<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>) than OpenCap's original enhancer (mean: 40.4<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>, max: 252.0<inline-formula><tex-math notation="LaTeX">^\circ</tex-math></inline-formula>). Conclusion: Our marker enhancer demonstrates both improved accuracy and generalizability across diverse movements. Significance: We integrated the marker enhancer into OpenCap, thereby offering its thousands of users more accurate measurements across a broader range of movements.