Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

• Published in: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 9848 - 9858
• Main Authors: Li, Rui, Fischer, Tobias, Segu, Mattia, Pollefeys, Marc, Van Gool, Luc, Tombari, Federico
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Attention mechanisms; Computer vision; Geometry; radiance field; Semantics; Shape; spatial attention; spatial context; Three-dimensional displays; vision-language model; Visualization
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52733.2024.00940

Recovering the 3D scene geometry from a single view is a fundamental yet ill-posed problem in computer vision. While classical depth estimation methods infer only a 2.5D scene representation limited to the image plane, recent approaches based on radiance fields reconstruct a full 3D representation. However, these methods still struggle with occluded regions since inferring geometry without visual observation requires (i) semantic knowledge of the surroundings, and (ii) reasoning about spatial context. We propose KYN, a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We introduce a vision-language modulation module to enrich point features with fine-grained semantic information. We aggregate point representations across the scene through a language-guided spatial attention mechanism to yield per-point density predictions aware of the 3D semantic context. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work. Project page: https://ruili3.github.io/kyn.