LISA: Learning Implicit Shape and Appearance of Hands

• Published in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 20501 - 20511
• Main Authors: Corona, Enric, Hodan, Tomas, Vo, Minh, Moreno-Noguer, Francesc, Sweeney, Chris, Newcombe, Richard, Ma, Lingni
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2022
• Subjects: Face and gestures; 3D from single images; Pose estimation and tracking; Fitting; Image color analysis; Pose estimation; Shape; Solid modeling; Surface reconstruction; Three-dimensional displays
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52688.2022.01988

This paper proposes a do-it-all neural model of human hands, named LISA. The model can capture accurate hand shape and appearance, generalize to arbitrary hand sub-jects, provide dense surface correspondences, be reconstructed from images in the wild, and can be easily an-imated. We train LISA by minimizing the shape and appearance losses on a large set of multi-view RGB image se-quences annotated with coarse 3D poses of the hand skele-ton. For a 3D point in the local hand coordinates, our model predicts the color and the signed distance with respect to each hand bone independently, and then combines the per-bone predictions using the predicted skinning weights. The shape, color, and pose representations are disentangled by design, enabling fine control of the selected hand param-eters. We experimentally demonstrate that LISA can ac-curately reconstruct a dynamic hand from monocular or multi-view sequences, achieving a noticeably higher qual-ity of reconstructed hand shapes compared to baseline approaches. Project page: https://www.iri.upc.edu/people/ecorona/lisa/.