3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction

• Published in: IEEE transactions on multimedia Vol. 25; pp. 3166 - 3179
• Main Authors: Chen, Shu-Yu, Lai, Yu-Kun, Xia, Shihong, Rosin, Paul L, Gao, Lin
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cameras; Communication; Eye movements; eye tracking; Face recognition; Faces; Gaze tracking; head-mounted display; Helmet mounted displays; Image reconstruction; real-time facial performance capture; Resists; Solid modeling; Three-dimensional displays; user interaction; Virtual reality
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3156820

With the rapid development of virtual reality (VR) technology, VR headsets, a.k.a. Head-Mounted Displays (HMDs), are widely available, allowing immersive 3D content to be viewed. A natural need for truly immersive VR is to allow bidirectional communication: the user should be able to interact with the virtual world using facial expressions and eye gaze, in addition to traditional means of interaction. The typical application scenario includes VR virtual conferencing and virtual roaming, where ideally users are able to see other users' expressions and have eye contact with them in the virtual world. In addition, eye gaze also provides a natural means of interaction with virtual objects. Despite significant achievements in recent years for reconstruction of 3D faces from RGB or RGB-D images, it remains a challenge to reliably capture and reconstruct 3D facial expressions including eye gaze when the user is wearing an HMD, because the majority of the face is occluded, especially those areas around the eyes which are essential for recognizing facial expressions and eye gaze. In this paper, we introduce a novel real-time system that is able to capture and reconstruct 3D faces wearing HMDs, and robustly recover eye gaze. We further propose a novel method to map eye gaze directions to the 3D virtual world, which provides a novel and useful interactive mode in VR. We compare our method with state-of-the-art techniques both qualitatively and quantitatively, and demonstrate the effectiveness of our system using live capture.