Facial Expression Re-targeting from a Single Character

Video retargeting for digital face animation is used in virtual reality, social media, gaming, movies, and video conference, aiming to animate avatars' facial expressions based on videos of human faces. The standard method to represent facial expressions for 3D characters is by blendshapes, a v...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Larey, Ariel, Asraf, Omri, Kelder, Adam, Wilf, Itzik, Kruzel, Ofer, Nati, Daniel
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 21.06.2023
Subjects
Animation
Avatars
Blinking
Datasets
Synthetic data
Videoconferencing
Virtual reality
Online AccessGet full text

Cover

More Information
Summary:Video retargeting for digital face animation is used in virtual reality, social media, gaming, movies, and video conference, aiming to animate avatars' facial expressions based on videos of human faces. The standard method to represent facial expressions for 3D characters is by blendshapes, a vector of weights representing the avatar's neutral shape and its variations under facial expressions, e.g., smile, puff, blinking. Datasets of paired frames with blendshape vectors are rare, and labeling can be laborious, time-consuming, and subjective. In this work, we developed an approach that handles the lack of appropriate datasets. Instead, we used a synthetic dataset of only one character. To generalize various characters, we re-represented each frame to face landmarks. We developed a unique deep-learning architecture that groups landmarks for each facial organ and connects them to relevant blendshape weights. Additionally, we incorporated complementary methods for facial expressions that landmarks did not represent well and gave special attention to eye expressions. We have demonstrated the superiority of our approach to previous research in qualitative and quantitative metrics. Our approach achieved a higher MOS of 68% and a lower MSE of 44.2% when tested on videos with various users and expressions.
ISSN:2331-8422