Investigating Personalization Methods in Text to Music Generation

In this work, we investigate the personalization of text-to-music diffusion models in a few-shot setting. Motivated by recent advances in the computer vision domain, we are the first to explore the combination of pre-trained text-to-audio diffusers with two established personalization methods. We ex...

Full description

Saved in:
Bibliographic Details
Published inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 1081 - 1085
Main Authors Plitsis, Manos, Kouzelis, Theodoros, Paraskevopoulos, Georgios, Katsouros, Vassilis, Panagakis, Yannis
Format Conference Proceeding
LanguageEnglish
Published IEEE 14.04.2024
Subjects
Codes
Computational modeling
Computer vision
diffusion
Measurement
personalization
Signal processing
System performance
text-to-music
Training
Online AccessGet full text

Cover

More Information
Summary:In this work, we investigate the personalization of text-to-music diffusion models in a few-shot setting. Motivated by recent advances in the computer vision domain, we are the first to explore the combination of pre-trained text-to-audio diffusers with two established personalization methods. We experiment with the effect of audio-specific data augmentation on the overall system performance and assess different training strategies. For evaluation, we construct a novel dataset with prompts and music clips. We consider both embedding-based and music-specific metrics for quantitative evaluation, as well as a user study for qualitative evaluation. Our analysis shows that similarity metrics are in accordance with user preferences and that current personalization approaches tend to learn rhythmic music concepts more easily than melody. The code, dataset, and example material of this study are open to the research community 1 .
ISSN:2379-190X
DOI:10.1109/ICASSP48485.2024.10446869