TrajectoryCNN: A New Spatio-Temporal Feature Learning Network for Human Motion Prediction

• Published in: IEEE transactions on circuits and systems for video technology Vol. 31; no. 6; pp. 2133 - 2146
• Main Authors: Liu, Xiaoli, Yin, Jianqin, Liu, Jin, Ding, Pengxiang, Liu, Jun, Liu, Huaping
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Benchmark testing; Biological system modeling; CNN; Computer vision; Correlation; Dynamics; Human motion; Human motion prediction; Joints (anatomy); Learning; Predictive models; Robotics; Robots; skeleton; spatio-temporal feature learning; Trajectory
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2020.3021409

Human motion prediction is an increasingly interesting topic in computer vision and robotics. In this paper, we propose a new end-to-end feedforward network, TrajectoryCNN, to predict future poses. Compared with the most existing methods, we introduce a new trajectory space and focus on modeling motion dynamics of the input sequence with coupled spatio-temporal features, dynamic local-global features, and global temporal co-occurrence features in the new space. Specifically, the coupled spatio-temporal features describe the spatial and temporal structural information hidden in a natural human motion sequence, which can be easily mined using CNN by simultaneously covering the spatial and temporal dimensions of the sequence with the convolutional filters. The dynamic local-global features encode different correlations among joint trajectories of human motion (i.e. strong correlations among joint trajectories of one part and weak correlations among joint trajectories of different parts), which can be captured by stacking multiple residual trajectory blocks and incorporating our skeletal representation. The global temporal co-occurrence features represent different importance of different input poses to mine the motion dynamics for predicting future poses, which can be obtained automatically by learning free parameters for each pose with our TrajectoryCNN. Finally, we predict future poses with the captured motion dynamic features in a non-recursive manner. Extensive experiments show that our method achieves state-of-the-art performance on five benchmarks (e.g. Human3.6M, CMU-Mocap, 3DPW, G3D, and FNTU), which demonstrates the effectiveness of our proposed method. The code is available at https://github.com/lily2lab/TrajectoryCNN.git .