PKU-DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 22530 - 22540
• Main Authors: Zheng, Xiaoyun, Liao, Liwei, Li, Xufeng, Jiao, Jianbo, Wang, Rongjie, Gao, Feng, Wang, Shiqi, Wang, Ronggang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 16.06.2024
• Subjects: Benchmark testing; Computer vision; Dynamics; Heuristic algorithms; Human reconstruction; Lighting; Multi-view videos; Neural radiance field; Novel view synthesis; Rendering (computer graphics)
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52733.2024.02126

High-quality human reconstruction and photo-realistic rendering of a dynamic scene is a long-standing problem in computer vision and graphics. Despite considerable ef-forts invested in developing various capture systems and re-construction algorithms, recent advancements still struggle with loose or oversized clothing and overly complex poses. In part, this is due to the challenges of acquiring high-quality human datasets. To facilitate the development of these fields, in this paper, we present PKU-DyMVHumans, a versatile human-centric dataset for high-fidelity reconstruction and rendering of dynamic human scenarios from dense multi-view videos. It comprises 8.2 million frames captured by more than 56 synchronized cameras across diverse scenarios. These sequences comprise 32 human subjects across 45 different scenarios, each with a high-detailed appearance and realistic human motion. Inspired by recent advancements in neural radiance field (NeRF)-based scene representations, we carefully set up an off-the-shelf framework that is easy to provide those state-of-the-art NeRF-based implementations and benchmark on PKU-DyMVHumans dataset. It is paving the way for various applications like fine-grained foreground/background de-composition, high-quality human reconstruction and photo-realistic novel view synthesis of a dynamic scene. Exten-sive studies are performed on the benchmark, demonstrating new observations and challenges that emerge from using such high-fidelity dynamic data. The project page and data is available at: https://pku-dymvhumans.github.io.