HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling

• Published in: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 16610 - 16620
• Main Authors: Attal, Benjamin, Huang, Jia-Bin, Richardt, Christian, Zollhofer, Michael, Kopf, Johannes, O'Toole, Matthew, Kim, Changil
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2023
• Subjects: 3D from multi-view and sensors; 6-DOF; Memory management; Pipelines; Rendering (computer graphics); Streaming media; Training; Visualization
• ISSN: 2575-7075
• DOI: 10.1109/CVPR52729.2023.01594

Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel-a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.