FLEX: Full-Body Grasping Without Full-Body Grasps

• Published in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 21179 - 21189
• Main Authors: Tendulkar, Purva, Suris, Didac, Vondrick, Carl
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2023
• Subjects: Avatars; body; Flexible printed circuits; gesture; Grasping; Humans: Face; movement; pose; Solid modeling; Three-dimensional displays; Training; Video games
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52729.2023.02029

Synthesizing 3D human avatars interacting realistically with a scene is an important problem with applications in AR/VR, video games, and robotics. Towards this goal, we address the task of generating a virtual human - hands and full body - grasping everyday objects. Existing methods approach this problem by collecting a 3D dataset of humans interacting with objects and training on this data. However, 1) these methods do not generalize to different object positions and orientations or to the presence of furniture in the scene, and 2) the diversity of their generated full-body poses is very limited. In this work, we address all the above challenges to generate realistic, diverse full-body grasps in everyday scenes without requiring any 3D full-body grasping data. Our key insight is to leverage the existence of both full-body pose and hand-grasping priors, composing them using 3D geometrical constraints to obtain full-body grasps. We empirically validate that these constraints can generate a variety of feasible human grasps that are superior to baselines both quantitatively and qualitatively. See our webpage for more details: flex.cs.columbia.edu.