An Open CAPT System for Prosody Practice: Practical Steps towards Multilingual Setup

• Published in: Languages (Basel) Vol. 9; no. 1; p. 27
• Main Authors: Blake, John, Bogach, Natalia, Kusakari, Akemi, Lezhenin, Iurii, Khaustova, Veronica, Xuan, Son Luu, Nguyen, Van Nhi, Pham, Nam Ba, Svechnikov, Roman, Ostapchuk, Andrey, Efimov, Dmitrei, Pyshkin, Evgeny
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.01.2024
• Subjects: Algorithms; Analysis; Artificial intelligence; CAPT; Cognitive style; Computer assisted language learning; Computer-assisted instruction; Feedback; Grammar, Comparative and general; iCALL; Intonation; Language acquisition; Language instruction; Linguistics; Mora; Multilingualism; multimodality; Phonology; Pitch; Pronunciation instruction; Prosody; Rhythm; Signal processing; Software; Sound waves; Speech; speech prosody; Technology application; Tonal languages; Japan
• ISSN: 2226-471X; 2226-471X
• DOI: 10.3390/languages9010027

This paper discusses the challenges posed in creating a Computer-Assisted Pronunciation Training (CAPT) environment for multiple languages. By selecting one language from each of three different language families, we show that a single environment may be tailored to cater for different target languages. We detail the challenges faced during the development of a multimodal CAPT environment comprising a toolkit that manages mobile applications using speech signal processing, visualization, and estimation algorithms. Since the applied underlying mathematical and phonological models, as well as the feedback production algorithms, are based on sound signal processing and modeling rather than on particular languages, the system is language-agnostic and serves as an open toolkit for developing phrasal intonation training exercises for an open selection of languages. However, it was necessary to tailor the CAPT environment to the language-specific particularities in the multilingual setups, especially the additional requirements for adequate and consistent speech evaluation and feedback production. In our work, we describe our response to the challenges in visualizing and segmenting recorded pitch signals and modeling the language melody and rhythm necessary for such a multilingual adaptation, particularly for tonal syllable-timed and mora-timed languages.