MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation

• Published in: IEEE transactions on biometrics, behavior, and identity science Vol. 3; no. 1; pp. 16 - 30
• Main Authors: Sarandi, Istvan, Linder, Timm, Arras, Kai Oliver, Leibe, Bastian
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: 3D human pose estimation; absolute human pose; Heating systems; Image reconstruction; Pose estimation; Post-production processing; Representations; Robustness; scale estimation; Space heating; Three-dimensional displays; Training; truncation; Two dimensional displays
• ISSN: 2637-6407; 2637-6407
• DOI: 10.1109/TBIOM.2020.3037257

Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust ( MeTRo ) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D. Our code is publicly available.<xref ref-type="fn" rid="fn1"> 1 1 https://vision.rwth-aachen.de/metrabs