ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields with Diffusion Models

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 20385 - 20395
• Main Authors: Shih, Meng-Li, Ma, Wei-Chiu, Boyice, Lorenzo, Holynski, Aleksander, Cole, Forrester, Curless, Brian, Kontkanen, Janne
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Computational modeling; Computer vision; Data collection; Diffusion models; Extrapolation; Neural radiance field; Neural Rendering; Novel view synthesis
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.01927

We propose ExtraNeRF, a novel method for extrapolating the range of views handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs to model scene-specific, fine-grained details, while capitalizing on diffusion models to extrapolate beyond our observed data. A key ingredient is to track visibility to determine what portions of the scene have not been observed, and focus on re-constructing those regions consistently with diffusion mod-els. Our primary contributions include a visibility-aware diffusion-based inpainting module that is fine-tuned on the input imagery, yielding an initial NeRF with moderate qual-ity (often blurry) inpainted regions, followed by a second diffusion model trained on the input imagery to consistently enhance, notably sharpen, the inpainted imagery from the first pass. We demonstrate high-quality results, extrapolating beyond a small number of (typically six or fewer) input views, effectively outpainting the NeRF as well as inpainting newly disoccluded regions inside the original viewing volume. We compare with related work both quantitatively and qualitatively and show significant gains over prior art.