Model-Based Approach for Measuring the Fairness in ASR

• Published in: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 6532 - 6536
• Main Authors: Liu, Zhe, Veliche, Irina-Elena, Peng, Fuchun
• Format: Conference Proceeding
• Language: English
• Published: IEEE 23.05.2022
• Subjects: Atmospheric measurements; Automatic speech recognition; Conferences; Error analysis; fairness; Measurement uncertainty; Particle measurements; Poisson regression; random effect; Signal processing; Sociology
• DOI: 10.1109/ICASSP43922.2022.9747654

The issue of fairness arises when the automatic speech recognition (ASR) systems do not perform equally well for all subgroups of the population. In any fairness measurement studies for ASR, the open questions of how to control the confounding factors, how to handle unobserved heterogeneity across speakers, and how to trace the source of any word error rate (WER) gap among different subgroups are especially important - if not appropriately accounted for, incorrect conclusions will be drawn. In this paper, we introduce mixed-effects Poisson regression to better measure and interpret any WER difference among subgroups of interest. Particularly, the presented method can effectively address the three problems raised above and is very flexible to use in practical disparity analyses. We demonstrate the validity of proposed model-based approach on both synthetic and real-world speech data.