Learning Music-Dance Representations Through Explicit-Implicit Rhythm Synchronization

• Published in: IEEE transactions on multimedia Vol. 26; pp. 8454 - 8463
• Main Authors: Yu, Jiashuo, Pu, Junfu, Cheng, Ying, Feng, Rui, Shan, Ying
• Format: Journal Article
• Language: English
• Published: IEEE 01.01.2024
• Subjects: Feature extraction; Humanities; Multimodal learning; music and dance; Rhythm; self-supervised learning; Synchronization; Task analysis; Videos; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2023.3303690

Although audio-visual representation has been proven to be applicable in many downstream tasks, the representation of dancing videos, which is more specific and always accompanied by music with complex auditory contents, remains challenging and uninvestigated. Considering the intrinsic alignment between the cadent movement of the dancer and music rhythm, we introduce MuDaR , a novel Music-Dance R epresentation learning framework to perform the synchronization of music and dance rhythms both in explicit and implicit ways. Specifically, we derive the dance rhythms based on visual appearance and motion cues inspired by the music rhythm analysis. Then the visual rhythms are temporally aligned with the music counterparts, which are extracted by the amplitude of sound intensity. Meanwhile, we exploit the implicit coherence of rhythms implied in audio and visual streams by contrastive learning. The model learns the joint embedding by predicting the temporal consistency between audio-visual pairs. The music-dance representation, together with the capability of detecting audio and visual rhythms, can further be applied to three downstream tasks: (a) dance classification, (b) music-dance retrieval, and (c) music-dance retargeting. Extensive experiments demonstrate that our proposed framework outperforms other self-supervised methods by a large margin.