Learning to Play Guitar with Robotic Hands

• Published in: Computer graphics forum Vol. 43; no. 8
• Main Authors: Luo, Chaoyi, Tang, Pengbin, Ma, Yuqi, Huang, Dongjin
• Format: Journal Article
• Language: English
• Published: Oxford Blackwell Publishing Ltd 01.12.2024
• Subjects: Computer graphics; Deep learning; End effectors; Fingers; Guitars; Hands; Human motion; Inverse kinematics; Motion capture; Neural networks; Performance evaluation; Qualitative analysis; Robot learning; Robotics; Tablature; Visual effects
• DOI: 10.1111/cgf.15166
• ISSN: 0167-7055

Playing the guitar is a dexterous human skill that poses significant challenges in computer graphics and robotics due to the precision required in finger positioning and coordination between hands. Current methods often rely on motion capture data to replicate specific guitar playing segments, which restricts the range of performances and demands intricate post‐processing. In this paper, we introduce a novel reinforcement learning model that can play the guitar using robotic hands, without the need for motion capture datasets, from input tablatures. To achieve this, we divide the simulation task for playing guitar into three stages. (a): for an input tablature, we first generate corresponding fingerings that align with human habits. (b): based on the generated fingerings as the guidance, we train a neural network for controlling the fingers of the left hand using deep reinforcement learning, and (c): we generate plucking movements for the right hand based on inverse kinematics according to the tablature. We evaluate our method by employing precision, recall, and F1 scores as quantitative metrics to thoroughly assess its performance in playing musical notes. In addition, we conduct qualitative analysis through user studies to evaluate the visual and auditory effects of guitar performance. The results demonstrate that our model excels in playing most moderately difficult and easier musical pieces, accurately playing nearly all notes.