Synthesis of Compositional Animations from Textual Descriptions

• Published in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 1376 - 1386
• Main Authors: Ghosh, Anindita, Cheema, Noshaba, Oguz, Cennet, Theobalt, Christian, Slusallek, Philipp
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.01.2021
• Subjects: Animation; Computer vision; Gestures and body pose; Manifolds; Motion pictures; Natural languages; Solid modeling; Three-dimensional displays; Vision + language
• ISSN: 2380-7504
• DOI: 10.1109/ICCV48922.2021.00143

"How can we animate 3D-characters from a movie script or move robots by simply telling them what we would like them to do?" "How unstructured and complex can we make a sentence and still generate plausible movements from it?" These are questions that need to be answered in the long-run, as the field is still in its infancy. Inspired by these problems, we present a new technique for generating compositional actions, which handles complex input sentences. Our output is a 3D pose sequence depicting the actions in the input sentence. We propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and 3D pose sequences corresponding to the given motion. We learn two manifold representations of the motion, one each for the upper body and the lower body movements. Our model can generate plausible pose sequences for short sentences describing single actions as well as long complex sentences describing multiple sequential and compositional actions. We evaluate our proposed model on the publicly available KIT Motion-Language Dataset containing 3D pose data with human-annotated sentences. Experimental results show that our model advances the state-of-the-art on text-based motion synthesis in objective evaluations by a margin of 50%. Qualitative evaluations based on a user study indicate that our synthesized motions are perceived to be the closest to the ground-truth motion captures for both short and compositional sentences.