Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. To train this network, we augment...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Li, Zhengqin, Shafiei, Mohammad, Ramamoorthi, Ravi, Sunkavalli, Kalyan, Chandraker, Manmohan
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 07.05.2019
Subjects
Augmented reality
Estimation
Lighting
Non-Lambertian surfaces
Reflectance
Rendering
Online AccessGet full text

Cover

More Information
Summary:We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. To train this network, we augment the SUNCG indoor scene dataset with real-world materials and render them with a fast, high-quality, physically-based GPU renderer to create a large-scale, photorealistic indoor dataset. Our inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -- allowing us to jointly reason about shape, lighting, and reflectance. Experiments show that our framework outperforms previous methods for estimating individual scene components, which also enables various novel applications for augmented reality, such as photorealistic object insertion and material editing. Code and data will be made publicly available.
ISSN:2331-8422