DEGAS: Detailed Expressions on Full-Body Gaussian Avatars

• Main Authors: Shao, Zhijing, Wang, Duotun, Tian, Qing-Yao, Yang, Yao-Dong, Meng, Hengyu, Cai, Zeyu, Dong, Bo, Zhang, Yu, Zhang, Kang, Wang, Zeyu
• Format: Journal Article
• Language: English
• Published: 20.08.2024
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Graphics
• DOI: 10.48550/arxiv.2408.10588

Although neural rendering has made significant advancements in creating lifelike, animatable full-body and head avatars, incorporating detailed expressions into full-body avatars remains largely unexplored. We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. Trained on multiview videos of a given subject, our method learns a conditional variational autoencoder that takes both the body motion and facial expression as driving signals to generate Gaussian maps in the UV layout. To drive the facial expressions, instead of the commonly used 3D Morphable Models (3DMMs) in 3D head avatars, we propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars. Leveraging the rendering capability of 3DGS and the rich expressiveness of the expression latent space, the learned avatars can be reenacted to reproduce photorealistic rendering images with subtle and accurate facial expressions. Experiments on an existing dataset and our newly proposed dataset of full-body talking avatars demonstrate the efficacy of our method. We also propose an audio-driven extension of our method with the help of 2D talking faces, opening new possibilities to interactive AI agents.