Cross-Lingual Neural Network Speech Synthesis Based on Multiple Embeddings

• Published in: International journal of interactive multimedia and artificial intelligence Vol. 7; no. 2; pp. 110 - 120
• Main Authors: Nosek, Tijana V, Suzic, Sinisa B, Pekar, Darko J, Obradovic, Radovan J, Secujski, Milan S, Delic, Vlado D
• Format: Journal Article
• Language: English
• Published: IMAI Software 01.12.2021; Universidad Internacional de La Rioja (UNIR)
• Subjects: artificial neural networks; cross-lingual; Neural networks; speech synthesis; vocoder; United Kingdom; Serbia; China; Spain
• ISSN: 1989-1660; 1989-1660
• DOI: 10.9781/ijimai.2021.11.005

The paper presents a novel architecture and method for speech synthesis in multiple languages, in voices of multiple speakers and in multiple speaking styles, even in cases when speech from a particular speaker in the target language was not present in the training data. The method is based on the application of neural network embedding to combinations of speaker and style IDs, but also to phones in particular phonetic contexts, without any prior linguistic knowledge on their phonetic properties. This enables the network not only to efficiently capture similarities and differences between speakers and speaking styles, but to establish appropriate relationships between phones belonging to different languages, and ultimately to produce synthetic speech in the voice of a certain speaker in a language that he/she has never spoken. The validity of the proposed approach has been confirmed through experiments with models trained on speech corpora of American English and Mexican Spanish. It has also been shown that the proposed approach supports the use of neural vocoders, i.e. that they are able to produce synthesized speech of good quality even in languages that they were not trained on. KEYWORDS Cross-lingual, Neural Networks, Speech Synthesis, Vocoder.