LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors

• Published in: IEEE transactions on visualization and computer graphics Vol. 29; no. 5; pp. 2337 - 2347
• Main Authors: Ren, Yiming, Zhao, Chengfeng, He, Yannan, Cong, Peishan, Liang, Han, Yu, Jingyi, Xu, Lan, Ma, Yuexin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.05.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Datasets; Global geometry; Human motion; Human motion capture; Inertial sensing devices; Laser radar; Lidar; Motion capture; Multisensor fusion; Optical sensors; Point cloud compression; sensor fusion; shape modeling; Three-dimensional displays; Trajectories; Trajectory; virtual reality
• ISSN: 1077-2626; 1941-0506
• DOI: 10.1109/TVCG.2023.3247088

We propose a multi-sensor fusion method for capturing challenging 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios, only using single LiDAR and 4 IMUs, which are set up conveniently and worn lightly. Specifically, to fully utilize the global geometry information captured by LiDAR and local dynamic motions captured by IMUs, we design a two-stage pose estimator in a coarse-to-fine manner, where point clouds provide the coarse body shape and IMU measurements optimize the local actions. Furthermore, considering the translation deviation caused by the view-dependent partial point cloud, we propose a pose-guided translation corrector. It predicts the offset between captured points and the real root locations, which makes the consecutive movements and trajectories more precise and natural. Moreover, we collect a LiDAR-IMU multi-modal mocap dataset, LIPD, with diverse human actions in long-range scenarios. Extensive quantitative and qualitative experiments on LIPD and other open datasets all demonstrate the capability of our approach for compelling motion capture in large-scale scenarios, which outperforms other methods by an obvious margin. We will release our code and captured dataset to stimulate future research.