Theme Transformer: Symbolic Music Generation With Theme-Conditioned Transformer

• Published in: IEEE transactions on multimedia Vol. 25; pp. 3495 - 3508
• Main Authors: Shih, Yi-Jen, Wu, Shih-Lun, Zalkow, Frank, Muller, Meinard, Yang, Yi-Hsuan
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Automatic symbolic music generation; Bars; Clustering; Coders; Conditioning; contrastive learning; Decoding; Deep learning; Encoders-Decoders; Music; parallel attention; positional encoding; Priming; Testing; theme retrieval; theme-conditioned generation; Training; Transformers
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3161851

Attention-based Transformer models have been increasingly employed for automatic music generation. To condition the generation process of such a model with a user-specified sequence, a popular approach is to take that conditioning sequence as a priming sequence and ask a Transformer decoder to generate a continuation. However, this prompt-based conditioning cannot guarantee that the conditioning sequence would develop or even simply repeat itself in the generated continuation. In this paper, we propose an alternative conditioning approach, called theme-based conditioning , that explicitly trains the Transformer to treat the conditioning sequence as a thematic material that has to manifest itself multiple times in its generation result. This is achieved with two main technical contributions. First, we propose a deep learning-based approach that uses contrastive representation learning and clustering to automatically retrieve thematic materials from music pieces in the training data. Second, we propose a novel gated parallel attention module to be used in a sequence-to-sequence (seq2seq) encoder/decoder architecture to more effectively account for a given conditioning thematic material in the generation process of the Transformer decoder. We report on objective and subjective evaluations of variants of the proposed Theme Transformer and the conventional prompt-based baseline, showing that our best model can generate, to some extent, polyphonic pop piano music with repetition and plausible variations of a given condition.