3D Human Scan With A Moving Event Camera

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) pp. 5586 - 5596
• Main Authors: Kohyama, Kai, Shiba, Shintaro, Aoki, Yoshimitsu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 17.06.2024
• Subjects: Accuracy; Computer vision; Event camera; Human mesh recovery; Human pose estimation; Pose estimation; Solid modeling; Three-dimensional displays; Virtual reality; Vision sensors
• ISSN: 2160-7516
• DOI: 10.1109/CVPRW63382.2024.00568

Capturing the 3D human body is one of the important tasks in computer vision with a wide range of applications such as virtual reality and sports analysis. However, conventional frame cameras are limited by their temporal resolution and dynamic range, which imposes constraints in real-world application setups. Event cameras have the advantages of high temporal resolution and high dynamic range (HDR), but the development of event-based methods is necessary to handle data with different characteristics. This paper proposes a novel event-based method for 3D pose estimation and human mesh recovery. Prior work on event-based human mesh recovery require frames (images) as well as event data. The proposed method solely relies on events; it carves 3D voxels by moving the event camera around a stationary body, reconstructs the human pose and mesh by attenuated rays, and fit statistical body models, preserving high-frequency details. The experimental results show that the proposed method outperforms conventional frame-based methods in the estimation accuracy of both pose and body mesh. We also demonstrate results in challenging situations where other frame-based methods suffer from motion blur. This is the first-of-its-kind to demonstrate event-only human mesh recovery, and we hope that it is the first step toward achieving robust and accurate 3D human body scanning from vision sensors.