LOLNeRF: Learn from One Look

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 1548 - 1557
• Main Authors: Rebain, Daniel, Matthews, Mark, Yi, Kwang Moo, Lagun, Dmitry, Tagliasacchi, Andrea
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2022
• Subjects: 3D from single images; Face and gestures; Image and video synthesis and generation; Pose estimation and tracking; Segmentation; Costs; Data models; Geometry; grouping and shape analysis; Vision + graphics; Shape; Solid modeling; Three-dimensional displays; Training
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52688.2022.00161

We present a method for learning a generative 3D model based on neural radiance fields, trained solely from data with only single views of each object. While generating realistic images is no longer a difficult task, producing the corresponding 3D structure such that they can be rendered from different views is non-trivial. We show that, unlike existing methods, one does not need multi-view data to achieve this goal. Specifically, we show that by reconstructing many images aligned to an approximate canonical pose with a single network conditioned on a shared latent space, you can learn a space of radiance fields that models shape and appearance for a class of objects. We demonstrate this by training models to reconstruct object categories using datasets that contain only one view of each subject without depth or geometry information. Our experiments show that we achieve state-of-the-art results in novel view synthesis and high-quality results for monocular depth prediction. https://lolnerf.github.io.