Learning a Contact Potential Field for Modeling the Hand-Object Interaction

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 46; no. 8; pp. 5645 - 5662
• Main Authors: Yang, Lixin, Zhan, Xinyu, Li, Kailin, Xu, Wenqiang, Zhang, Junming, Li, Jiefeng, Lu, Cewu
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Apexes; Contact modeling; Contact potentials; Contacts; Estimation; Grasping; grasping pose generation; hand-object pose estimation; Image reconstruction; Mass-spring systems; Pose estimation; Potential fields; Semantics; Task analysis; Three-dimensional displays
• ISSN: 0162-8828; 1939-3539; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2024.3372102

Estimating and synthesizing the hand's manipulation of objects is central to understanding human behaviour. To accurately model the interaction between the hand and object (referred to as the "hand-object"), we must not only focus on the pose of the hand and object, but also consider the contact between them. This contact provides valuable information for generating semantically and physically plausible grasps. In this paper, we propose an explicit contact representation called C ontact P otential F ield (CPF). In CPF, we model the contact between a pair of hand-object vertices as a spring-mass system. This system encodes the distance of the pair, as well as a likelihood of that contact being stable. Therefore, the system of multiple extended and compressed springs forms an elastic potential field with minimal energy at the optimal grasp position. We apply CPF to two relevant tasks, namely, hand-object pose estimation and grasping pose generation. Extensive experiments on the two challenging tasks and three commonly used datasets have demonstrated that our method can achieve state-of-the-art in several reconstruction metrics, allowing us to produce more physically plausible hand-object poses even when the ground-truth exhibits severe interpenetration or disjointedness.