Indoor 3D Human Trajectory Reconstruction Using Surveillance Camera Videos and Point Clouds

• Published in: IEEE transactions on circuits and systems for video technology Vol. 32; no. 4; pp. 2482 - 2495
• Main Authors: Dai, Yudi, Wen, Chenglu, Wu, Hai, Guo, Yulan, Chen, Longbiao, Wang, Cheng
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D trajectory; Cameras; Constraint modelling; Cooperation; Datasets; Geometric constraints; Human activity recognition; Human trajectory reconstruction; image depth regression; Image reconstruction; Modules; Occlusion; Performance evaluation; point cloud; Pose estimation; Regression; Surveillance; surveillance video; Three dimensional models; Three-dimensional displays; Trajectory; Trajectory analysis; trajectory optimization; Video; Videos
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2021.3081591

3D human trajectory reconstruction in an indoor scene is critical in various applications, such as indoor navigation and human activity recognition. This task is challenging due to occlusion and clutters of indoor scenes, flexible human body joints, and severe lack of relevant datasets. Although several methods have been proposed to reconstruct a 3D human trajectory, they either can recover only 2D positions or require human initiative cooperation. In this paper, we propose a novel framework for 3D human trajectory reconstruction in an indoor scene using monocular surveillance videos and static point clouds without any initiative cooperation. The proposed framework consists of three modules: 3D pose estimation, depth regression, and trajectory reconstruction. We first estimate 3D pose from videos. Especially, we reconstruct a half-body 3D pose to deal with the occlusion problem. Then, we propose a depth regression approach to iteratively regress the depth of a 3D pose. Unlike data-driven approaches, our depth regression approach does not require training data and can be integrated into any 3D pose model. Finally, we exploit the geometric constraints from the point cloud to optimize the 3D trajectory. We evaluated the 3D pose estimation and depth regression modules on the H3.6M datasets. Due to the lack of evaluation datasets, we also built a trajectory dataset to evaluate the trajectory reconstruction performance. Empirical evaluation shows that our framework achieves accurate trajectory reconstruction results on real-world videos.