Rhythm is a Dancer: Music-Driven Motion Synthesis With Global Structure

• Published in: IEEE transactions on visualization and computer graphics Vol. 29; no. 8; pp. 3519 - 3534
• Main Authors: Aristidou, Andreas, Yiannakidis, Anastasios, Aberman, Kfir, Cohen-Or, Daniel, Shamir, Ariel, Chrysanthou, Yiorgos
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Aerospace electronics; Animation; Auditory Perception; Choreography; Computer Graphics; Computer science; Dance; Dancing; global structure consistency; Human motion; Humans; motion motifs; motion signatures; Movement; Music; music-driven; Rhythm; Skeleton; Synchronization; Synthesis; Task analysis
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2022.3163676

Synthesizing human motion with a global structure, such as a choreography, is a challenging task. Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion. In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre. In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat. Our music-driven dance synthesis framework is a hierarchical system that consists of three levels: pose , motif , and choreography . The pose level consists of an LSTM component that generates temporally coherent sequences of poses. The motif level guides sets of consecutive poses to form a movement that belongs to a specific distribution using a novel motion perceptual-loss . And the choreography level selects the order of the performed movements and drives the system to follow the global structure of a dance genre. Our results demonstrate the effectiveness of our music-driven framework to generate natural and consistent movements on various dance types, having control over the content of the synthesized motions, and respecting the overall structure of the dance.