Evaluating English language learners' conversations: Man vs. Machine

• Published in: Computer assisted language learning Vol. 32; no. 4; pp. 398 - 417
• Main Authors: Forsyth, Carol M., Luce, Christine, Zapata-Rivera, Diego, Jackson, G. Tanner, Evanini, Keelan, So, Youngsoon
• Format: Journal Article
• Language: English
• Published: Oxford Routledge 04.05.2019; Taylor & Francis; Taylor & Francis Ltd
• Subjects: Between-subjects design; Computer Assisted Testing; Computer Software; Conversation; conversation-based assessment; Discourse Analysis; Educational Assessment; Elementary School Students; English; English (Second Language); English as a second language; English as a second language learning; English language; English Language Learners; English language learning; English Learners; Feasibility studies; formative assessment; Formative Evaluation; Grade 3; Grade 4; Grade 5; Interpersonal Communication; Language Processing; Language Usage; Natural language; Second Language Learning; Students
• ISSN: 0958-8221; 1744-3210
• DOI: 10.1080/09588221.2018.1517126

This study investigated a new conversation-based assessment for English language learners. In this assessment, students converse with animated agents in natural language conversations to assess various aspects of English language learning. In a between subjects design, 31 students (N = 31) were asked questions by either the newly created system or human interviewers. Results revealed that there was no difference in answers in regards to impacting the constructs under investigation. However, there were other differences including higher word count and potentially more complex language usage when speaking to a human. Furthermore, students liked interacting with the system. These results suggest that the newly created system can assess students' English capabilities comparably to human interviewers and implementing such a formative assessment is feasible.