Music conditioned 2D hand gesture dance generation with HGS

• Published in: Computer animation and virtual worlds Vol. 35; no. 1
• Main Authors: Zhou, Dian, Liu, Shiguang, Xu, Qing
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.01.2024
• Subjects: Audio data; Dance; dance motion synthesis; digital humans; hand gesture dance; multimodal; pose generation
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.2211

In recent years, the short video industry is booming. However, there are still many difficulties in the action generation of virtual characters. We observed that on the short video social platform, “hand gesture dance” is a very popular short video form. However, its development is limited by the professionalism of choreography. In order to solve these problems, we propose an intelligent choreography framework, which can generate new gesture sequences for unseen audio based on pairing data in the database. Our framework adopts multimodal method and obtains excellent results. In additional, we collected and produced the first and largest pair labeled hand gesture dance data set. Various experiments showed that our results not only generate smooth and rich action sequences, but also collect some semantic information contained in the audio. We propose an intelligent choreography framework, which can generate new gesture sequences for unseen audio based on pairing data in the database. Our framework adopts multimodal method and obtains excellent results. In additional, we collected and produced the first and largest pair labeled hand gesture dance data set. Various experiments showed that our results not only generate smooth and rich action sequences, but also collect some semantic information contained in the audio.