Bidirectional temporal feature for 3D human pose and shape estimation from a video

• Published in: Computer animation and virtual worlds Vol. 34; no. 3-4
• Main Authors: Sun, Libo, Tang, Ting, Qu, Yuke, Qin, Wenhu
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01.05.2023; Wiley Subscription Services, Inc
• Subjects: Bi‐LSTM; Datasets; Discriminators; Estimation; Human motion; human pose and shape estimation; Spatial data; Three dimensional motion; transformer
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2187

3D human pose and shape estimation is the foundation of analyzing human motion. However, estimating accurate and temporally consistent 3D human motion from a video remains a challenge. By now, most of the video‐based methods for estimating 3D human pose and shape rely on unidirectional temporal features and lack more comprehensive information. To solve this problem, we propose a novel model “bidirectional temporal feature for human motion recovery” (BTMR), which consists of a human motion generator and a discriminator. The transformer‐based generator effectively captures the forward and reverse temporal features to enhance the temporal correlation between frames and reduces the loss of spatial information. The motion discriminator based on Bi‐LSTM can distinguish whether the generated pose sequences are consistent with the realistic sequences of the AMASS dataset. In the process of continuous generation and discrimination, the model can learn more realistic and accurate poses. We evaluate our BTMR on 3DPW and MPI‐INF‐3DHP datasets. Without the training set of 3DPW, BTMR outperforms VIBE by 2.4 mm and 14.9 mm/s2 in PA‐MPJPE and Accel metrics and outperforms TCMR by 1.7 mm in PA‐MPJPE metric on 3DPW. The results demonstrate that our BTMR produces better accurate and temporal consistent 3D human motion. Our bidirectional temporal feature for human motion recovery improves both temporal consistency and accuracy for estimating human motion from a video. It also helps to solve the problem of abnormal pose estimation for complex human motion.