SpiRit-LM : Interleaved Spoken and Written Language Model

• Published in: Transactions of the Association for Computational Linguistics Vol. 13; pp. 30 - 52
• Main Authors: Nguyen, Tu Anh, Muller, Benjamin, Yu, Bokai, Costa-jussa, Marta R., Elbayad, Maha, Popuri, Sravya, Ropers, Christophe, Duquenne, Paul-Ambroise, Algayres, Robin, Mavlyutov, Ruslan, Gat, Itai, Williamson, Mary, Synnaeve, Gabriel, Pino, Juan, Sagot, Benoît, Dupoux, Emmanuel
• Format: Journal Article
• Language: English
• Published: 255 Main Street, 9th Floor, Cambridge, Massachusetts 02142, USA MIT Press 07.01.2025; MIT Press Journals, The; The MIT Press
• Subjects: Computation and Language; Computational linguistics; Computer Science; Inference; Language modeling; Parallel corpora; Phonetics; Semantics; Speech; Speech recognition; Written language
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00728

We introduce , a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level method using a small automatically curated speech-text parallel corpus. comes in two versions: a version that uses speech phonetic units (HuBERT) and an version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that can learn new tasks in a few-shot fashion across modalities (i.e., ASR, TTS, Speech Classification). We make available model weights and inference code.