Laser: Latent Set Representations for 3D Generative Modeling

• Main Authors: Moreno, Pol, Kosiorek, Adam R, Strathmann, Heiko, Zoran, Daniel, Schneider, Rosalia G, Winckler, Björn, Markeeva, Larisa, Weber, Théophane, Rezende, Danilo J
• Format: Journal Article
• Language: English
• Published: 13.01.2023
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2301.05747

NeRF provides unparalleled fidelity of novel view synthesis: rendering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover a scene, which limits its applicability. While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts. We introduce Laser-NV: a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows. Similarly to previous amortized approaches, Laser-NV learns structure from multiple scenes and is capable of fast, feed-forward inference from few views. To encourage higher rendering fidelity and consistency with observed views, Laser-NV further incorporates a geometry-informed attention mechanism over the observed views. Laser-NV further produces diverse and plausible completions of occluded parts of a scene while remaining consistent with observations. Laser-NV shows state-of-the-art novel-view synthesis quality when evaluated on ShapeNet and on a novel simulated City dataset, which features high uncertainty in the unobserved regions of the scene.