ExpressGesture: Expressive gesture generation from speech through database matching

• Published in: Computer animation and virtual worlds Vol. 32; no. 3-4
• Main Authors: Ferstl, Ylva, Neff, Michael, McDonnell, Rachel
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.06.2021
• Subjects: Acceleration; computer animation; conversational agents; expressive agents; gesture generation; Gesture recognition; Machine learning; Motion capture; motion matching; Parameters; perception; Sequences; Speech
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2016

Co‐speech gestures are a vital ingredient in making virtual agents more human‐like and engaging. Automatically generated gestures based on speech‐input often lack realistic and defined gesture form. We present a database‐driven approach guaranteeing defined gesture form. We built a large corpus of over 23,000 motion‐captured co‐speech gestures and select individual gestures based on expressive gesture characteristics that can be estimated from speech audio. The expressive parameters are gesture velocity and acceleration, gesture size, arm swivel, and finger extension. Individual, parameter‐matched gestures are then combined into animated sequences. We evaluate our gesture generation system in two perceptual studies. The first study compares our method to the ground truth gestures as well as mismatched gestures. The second study compares our method to five current generative machine learning models. Our method outperformed mismatched gesture selection in the first study and showed competitive performance in the second. We present a system for automatic gesture generation from speech audio, together with a database of over 23,000 motion‐captured gestures. Using a hybrid approach of machine learning and database‐sampling, our system guarantees defined gesture form. Our system selects individual gestures based on expressive gesture characteristics that can be estimated from speech audio and then combines selected gestures into animated sequences. Our method outperforms mismatched gesture selection and shows competitive performance to current generative machine learning models.