BACH: Bi‐Stage Data‐Driven Piano Performance Animation for Controllable Hand Motion

• Published in: Computer animation and virtual worlds Vol. 36; no. 3
• Main Authors: Jiao, Jihui, Zeng, Rui, Dai, Ju, Pan, Junjun
• Format: Journal Article
• Language: English
• Published: Hoboken, USA John Wiley & Sons, Inc 01.05.2025; Wiley Subscription Services, Inc
• Subjects: Animation; Controllability; deep learning; Laplace transforms; Musical scores; music‐driven motion; piano performance animation; Pianos; Virtual humans
• ISSN: 1546-4261; 1546-427X
• DOI: 10.1002/cav.70044

ABSTRACT This paper presents a novel framework for generating piano performance animations using a two‐stage deep learning model. By using discrete musical score data, the framework transforms sparse control signals into continuous, natural hand motions. Specifically, in the first stage, by incorporating musical temporal context, the keyframe predictor is leveraged to learn keyframe motion guidance. Meanwhile, the second stage synthesizes smooth transitions between these keyframes via an inter‐frame sequence generator. Additionally, a Laplacian operator‐based motion retargeting technique is introduced, ensuring that the generated animations can be adapted to different digital human models. We demonstrate the effectiveness of the system through an audiovisual multimedia application. Our approach provides an efficient, scalable method for generating realistic piano animations and holds promise for broader applications in animation tasks driven by sparse control signals. Illustration of our animation generation pipeline.