The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

• Published in: Transactions of the Association for Computational Linguistics Vol. 10; pp. 522 - 538
• Main Authors: Goyal, Naman, Gao, Cynthia, Chaudhary, Vishrav, Chen, Peng-Jen, Wenzek, Guillaume, Ju, Da, Krishnan, Sanjana, Ranzato, Marc’Aurelio, Guzmán, Francisco, Fan, Angela
• Format: Journal Article
• Language: English
• Published: One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA MIT Press 04.05.2022; MIT Press Journals, The; The MIT Press
• Subjects: Benchmarks; Datasets; Interpreters; Language; Languages; Linguistics; Machine translation; Multilingualism; Quality assessment; Sentences; Translations; Translators
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00474

One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are fully aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.