Machine Learning–Driven Language Assessment

• Published in: Transactions of the Association for Computational Linguistics Vol. 8; pp. 247 - 263
• Main Authors: Settles, Burr, T. LaFlair, Geoffrey, Hagiwara, Masato
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.01.2020; MIT Press Journals, The; The MIT Press
• Subjects: Achievement tests; Assessments; English as a second language tests; English language; English proficiency; Human subjects; Item response theory; Language; Language assessment; Linguistics; Machine learning; Measures; Natural language processing; Quantitative psychology; Rasch model; Research subjects; Test validity and reliability
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00310

We describe a method for rapidly creating language proficiency assessments, and provide experimental evidence that such tests can be valid, reliable, and secure. Our approach is the first to use machine learning and natural language processing to induce proficiency scales based on a given standard, and then use linguistic models to estimate item difficulty directly for computer-adaptive testing. This alleviates the need for expensive pilot testing with human subjects. We used these methods to develop an online proficiency exam called the Duolingo English Test, and demonstrate that its scores align significantly with other high-stakes English assessments. Furthermore, our approach produces test scores that are highly reliable, while generating item banks large enough to satisfy security requirements.