SC-CNN: Effective Speaker Conditioning Method for Zero-Shot Multi-Speaker Text-to-Speech Systems

• Published in: IEEE signal processing letters Vol. 30; pp. 1 - 5
• Main Authors: Yoon, Hyungchan, Kim, Changhwan, Um, Seyun, Yoon, Hyun-Wook, Kang, Hong-Goo
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Bias; Conditioning; Context; Convolution; Convolutional neural networks; generalization; Kernel; Kernels; multi-speaker; Phonemes; Phonetics; Predictive models; Speech recognition; style transfer; System effectiveness; Task analysis; Text-to-speech; Training; zero-shot
• ISSN: 1070-9908; 1558-2361
• DOI: 10.1109/LSP.2023.3277786

This letter proposes an effective speaker-conditioning method that is applicable to zero-shot multi-speaker text-to-speech (ZSM-TTS) systems. Based on the inductive bias in the speech generation task, in which local context information in text/phoneme sequences heavily affect the speaker characteristics of the output speech, we propose a Speaker-Conditional Convolutional Neural Network (SC-CNN) for the ZSM-TTS task. SC-CNN first predicts convolutional kernels from each learned speaker embedding, then applies 1-D convolutions to phoneme sequences with the predicted kernels. It utilizes the aforementioned inductive bias and effectively models the characteristic of speech by providing the speaker-specific local context in phonetic domain. We also build both FastSpeech2 and VITS-based ZSM-TTS systems to verify its superiority over conventional speaker conditioning methods. The results confirm that the models with SC-CNN outperform the recent ZSM-TTS models in terms of both subjective and objective measurements.