Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 20331 - 20341
• Main Authors: Yang, Ziyi, Gao, Xinyu, Zhou, Wen, Jiao, Shaohui, Zhang, Yuqing, Jin, Xiaogang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: 3D Gaussian Splatting; Annealing; Deformable models; Dynamic reconstruction; Interpolation; Smoothing methods; Solid modeling; Three-dimensional displays; Training
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.01922

Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction. Nonetheless, cutting-edge dynamic neural rendering methods rely heavily on these implicit representations, which frequently struggle to capture the intricate details of objects in the scene. Furthermore, implicit methods have difficulty achieving real-time rendering in general dynamic scenes, limiting their use in a variety of tasks. To address the issues, we propose a deformable 3D Gaussians splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space with a deformation field to model monocular dynamic scenes. We also introduce an annealing smoothing training mechanism with no extra overhead, which can mitigate the impact of inaccurate poses on the smoothness of time interpolation tasks in real-world scenes. Through a differential Gaussian rasterizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed. Experiments show that our method outperforms existing methods significantly in terms of both rendering quality and speed, making it well-suited for tasks such as novel-view synthesis, time interpolation, and real-time rendering. Our code is available at https://github.com/ingra14m/Deformable-3D-Gaussians.