Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20654 - 20664
• Main Authors: Lu, Tao, Yu, Mulin, Xu, Linning, Xiangli, Yuanbo, Wang, Limin, Lin, Dahua, Dai, Bo
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Computational modeling; Computer vision; Gaussian Splatting; Lighting; Neural Rendering; Rendering (computer graphics); Semantics; Three-dimensional displays; Training
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.01952

Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications. The recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed combining the benefits of both primitive-based representations and volumetric representations. However, it often leads to heavily redundant Gaussians that try to fit every training view, neglecting the underlying scene ge-ometry. Consequently, the resulting model becomes less robust to significant view changes, texture-less area and lighting effects. We introduce Scaffold-GS, which uses an-chor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum. Anchor growing and pruning strategies are developed based on the importance of neural Gaussians to reliably improve the scene cover-age. We show that our method effectively reduces redun-dant Gaussians while delivering high-quality rendering. We also demonstrates an enhanced capability to accommodate scenes with varying levels-of-detail and view-dependent ob-servations, without sacrificing the rendering speed. Project page: https://city-super.github.iolscaffold-gsl.