Interaction With Virtual Objects Using Human Pose and Shape Estimation - Son Nguyen, Hong; Cheong, DaEun; Chalmers, Andrew; Gon Kim, Myoung; Rhee, Taehyun; Han, JungHyun (Computer animation and virtual worlds, 2025)
ABSTRACT In this article, we propose an AR system that facilitates a user's natural interaction with virtual objects in an augmented reality environment. The system consists of three modules: human pose and shape estimation, camera‐space calibration, and physics simulation. The first module estimates a user's 3D pose and shape from a single RGB video stream, thereby reducing the system setup cost and broadening potential applications. The camera‐space calibration module estimates the user's camera‐space position to align the user with the input RGB image. The physics simulation enables seamless and physically natural interaction with virtual objects. Two prototyping applications built upon the system prove an enhancement in the quality of interaction, fostering a more immersive and intuitive user experience. Our AR system is composed of three modules that facilitate interactions between a real human and a virtual objects. In this AR boxing game, the user is interacting with the virtual punching bag.