Boosting Monocular 3D Human Pose Estimation With Part Aware Attention

• Published in: IEEE transactions on image processing Vol. 31; pp. 4278 - 4291
• Main Authors: Xue, Youze, Chen, Jiansheng, Gu, Xiangming, Ma, Huimin, Ma, Hongbing
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D human pose estimation; Convolution; Correlation; Dictionaries; dictionary attention; Human motion; Modules; part aware attention; Pose estimation; Skeleton; temporal attention; Three-dimensional displays; Transformers
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2022.3182269

Monocular 3D human pose estimation is challenging due to depth ambiguity. Convolution-based and Graph-Convolution-based methods have been developed to extract 3D information from temporal cues in motion videos. Typically, in the lifting-based methods, most recent works adopt the transformer to model the temporal relationship of 2D keypoint sequences. These previous works usually consider all the joints of a skeleton as a whole and then calculate the temporal attention based on the overall characteristics of the skeleton. Nevertheless, the human skeleton exhibits obvious part-wise inconsistency of motion patterns. It is therefore more appropriate to consider each part's temporal behaviors separately. To deal with such part-wise motion inconsistency, we propose the Part Aware Temporal Attention module to extract the temporal dependency of each part separately. Moreover, the conventional attention mechanism in 3D pose estimation usually calculates attention within a short time interval. This indicates that only the correlation within the temporal context is considered. Whereas, we find that the part-wise structure of the human skeleton is repeating across different periods, actions, and even subjects. Therefore, the part-wise correlation at a distance can be utilized to further boost 3D pose estimation. We thus propose the Part Aware Dictionary Attention module to calculate the attention for the part-wise features of input in a dictionary, which contains multiple 3D skeletons sampled from the training set. Extensive experimental results show that our proposed part aware attention mechanism helps a transformer-based model to achieve state-of-the-art 3D pose estimation performance on two widely used public datasets. The codes and the trained models are released at https://github.com/thuxyz19/3D-HPE-PAA .