GPS-Gaussian: Generalizable Pixel-Wise 3D Gaussian Splatting for Real-Time Human Novel View Synthesis

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 19680 - 19690
• Main Authors: Zheng, Shunyuan, Zhou, Boyao, Shao, Ruizhi, Liu, Boning, Zhang, Shengping, Nie, Liqiang, Liu, Yebin
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Cameras; Computer vision; Estimation; Graphics processing units; Pipelines; Rendering (computer graphics); Three-dimensional displays
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.01861

We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting. Unlike the original Gaussian Splatting or neural implicit rendering methods that necessitate per-subject optimizations, we introduce Gaussian parameter maps defined on the source views and regress directly Gaussian Splatting properties for instant novel view synthesis without any fine-tuning or optimization. To this end, we train our Gaussian parameter regression module on a large amount of human scan data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. The proposed framework is fully differentiable and experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed. The code is available at https://github.com/aipixel/GPS-Gaussian.