Interaction With Virtual Objects Using Human Pose and Shape Estimation

• Published in: Computer animation and virtual worlds Vol. 36; no. 3
• Main Authors: Son Nguyen, Hong, Cheong, DaEun, Chalmers, Andrew, Gon Kim, Myoung, Rhee, Taehyun, Han, JungHyun
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01.05.2025; Wiley Subscription Services, Inc
• Subjects: Augmented reality; Calibration; Cameras; Estimates; full‐body interaction; Modules; pose and shape estimation; Pose estimation; Prototyping; User experience; Video data
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.70046

ABSTRACT In this article, we propose an AR system that facilitates a user's natural interaction with virtual objects in an augmented reality environment. The system consists of three modules: human pose and shape estimation, camera‐space calibration, and physics simulation. The first module estimates a user's 3D pose and shape from a single RGB video stream, thereby reducing the system setup cost and broadening potential applications. The camera‐space calibration module estimates the user's camera‐space position to align the user with the input RGB image. The physics simulation enables seamless and physically natural interaction with virtual objects. Two prototyping applications built upon the system prove an enhancement in the quality of interaction, fostering a more immersive and intuitive user experience. Our AR system is composed of three modules that facilitate interactions between a real human and a virtual objects. In this AR boxing game, the user is interacting with the virtual punching bag.