Point-NeRF: Point-based Neural Radiance Fields

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 5428 - 5438
• Main Authors: Xu, Qiangeng, Xu, Zexiang, Philip, Julien, Bi, Sai, Shu, Zhixin, Sunkavalli, Kalyan, Neumann, Ulrich
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2022
• Subjects: 3D from multi-view and sensors; Computational photography; Image and video synthesis and generation; Pipelines; Point cloud compression; Rendering (computer graphics); Solid modeling; Surface reconstruction; Three-dimensional displays; Training
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52688.2022.00536

Volumetric neural rendering methods like NeRF [34] generate high-quality view synthesis results but are optimized per-scene leading to prohibitive reconstruction time. On the other hand, deep multi-view stereo methods can quickly reconstruct scene geometry via direct network inference. Point-NeRF combines the advantages of these two approaches by using neural 3D point clouds, with associated neural features, to model a radiance field. Point-NeRF can be rendered efficiently by aggregating neural point features near scene surfaces, in a ray marching-based rendering pipeline. Moreover, Point-NeRF can be initialized via direct inference of a pre-trained deep network to produce a neural point cloud; this point cloud can be finetuned to surpass the visual quality of NeRF with 30× faster training time. Point-NeRF can be combined with other 3D re-construction methods and handles the errors and outliers in such methods via a novel pruning and growing mechanism.