A Probabilistically-Oriented Analysis of the Performance of ASR Systems for Brazilian Radios and TVs

• Published in: Intelligent Systems Vol. 13654; pp. 169 - 180
• Main Authors: de Azevedo, Diego Marques, Rodrigues, Guilherme Souza, Ladeira, Marcelo
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2022; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Audimus.Media; Brazilian Portuguese; GLM; Google Speech-to-Text; Kaldi; Microsoft Azure Speech; Speech recognition; Text corpus; Wav2vec 2.0
• ISBN: 3031216881; 9783031216886
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-21689-3_13

With the use of neural network-based technologies, Automatic Speech Recognition (ASR) systems for Brazilian Portuguese (BP) have shown great progress in the last few years. Several state-of-art results were achieved by open-source end-to-end models, such as the Kaldi toolkit and the Wav2vec 2.0. Alternative commercial tools are also available, including the Google and Microsoft speech to text APIs and the Audimus System of VoiceInteraction. We analyse the relative performance of such tools – in terms of the so-called Word Error Rate (WER) – when transcribing audio recordings from Brazilian radio and TV channels. A generalized linear model (GLM) is designed to stochastically describe the relationship between some of the audio’s properties (e.g. file format and audio duration) and the resulting WER, for each method under consideration. Among other uses, such strategy enables the analysis of local performances, indicating not only which tool performs better, but when exactly it is expected to do so. This, in turn, could be used to design an optimized system composed of several transcribers. The data generated for conducting this experiment and the scripts used to produce the stochastic model are public available.