Generative RGB-D Face Completion for Head-Mounted Display Removal

Head-mounted displays (HMDs) are an essential display device for the observation of virtual reality (VR) environments. However, HMDs obstruct external capturing methods from recording the user's upper face. This severely impacts social VR applications, such as teleconferencing, which commonly r...

Full description

Saved in:
Bibliographic Details
Published in2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) pp. 109 - 116
Main Authors Numan, Nels, Haar, Frank ter, Cesar, Pablo
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.03.2021
Subjects
Artificial intelligence
Computer vision
Computing methodologies
Conferences
Head-mounted displays
Human computer interaction (HCI)
Human-centered computing
Image color analysis
Interaction paradigms
Reconstruction
Resists
Three-dimensional displays
User interfaces
Virtual reality
Online AccessGet full text
DOI10.1109/VRW52623.2021.00028

Cover

More Information
Summary:Head-mounted displays (HMDs) are an essential display device for the observation of virtual reality (VR) environments. However, HMDs obstruct external capturing methods from recording the user's upper face. This severely impacts social VR applications, such as teleconferencing, which commonly rely on external RGB-D sensors to capture a volumetric representation of the user. In this paper, we introduce an HMD removal framework based on generative adversarial networks (GANs), capable of jointly filling in missing color and depth data in RGB-D face images. Our framework includes an RGB-based identity loss function for identity preservation and several components aimed at surface reproduction. Our results demonstrate that our framework is able to remove HMDs from synthetic RGB-D face images while preserving the subject's identity.
DOI:10.1109/VRW52623.2021.00028