Monocular Reconstruction of Neural Face Reflectance Fields

• Published in: arXiv.org
• Main Authors: Mallikarjun, B R, Tewari, Ayush, Tae-Hyun Oh, Weyrich, Tim, Bickel, Bernd, Seidel, Hans-Peter, Pfister, Hanspeter, Matusik, Wojciech, Elgharib, Mohamed, Theobalt, Christian
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 24.08.2020
• Subjects: Illumination; Image reconstruction; Light; Reflectance; Representations; Self shadowing; Shadows; Specular reflection
• ISSN: 2331-8422

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance as higher-order global illumination effects and self-shadowing are not modeled. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage training dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods, in terms of photorealism due to better capturing of physical premitives, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.