PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes

• Published in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 18541 - 18550
• Main Authors: Yeh, Yu-Ying, Li, Zhengqin, Hold-Geoffroy, Yannick, Zhu, Rui, Xu, Zexiang, Hasan, Milos, Sunkavalli, Kalyan, Chandraker, Manmohan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2022
• Subjects: Geometry; Layout; Lighting; Photorealism; Reconstruction algorithms; Solid modeling; Three-dimensional displays; Vision + graphics; Physics-based vision and shape-from-X
• DOI: 10.1109/CVPR52688.2022.01801

Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene 1 1 Code: https://github.com/ViLab-UCSD/PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.