Multicraft: A Multimodal Interface for Supporting and Studying Learning in Minecraft

• Published in: HCI in Games: Serious and Immersive Games Vol. 12790; pp. 113 - 131
• Main Authors: Worsley, Marcelo, Mendoza Tudares, Kevin, Mwiti, Timothy, Zhen, Mitchell, Jiang, Marc
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Constructionism; Data mining; Games; Spatial reasoning
• ISBN: 3030774139; 9783030774134
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-77414-1_10

In this paper, we present work on bringing multimodal interaction to Minecraft. The platform, Multicraft, incorporates speech-based input, eye tracking, and natural language understanding to facilitate more equitable gameplay in Minecraft. We tested the platform with elementary, middle school students and college students through a collection of studies. Students found each of the provided modalities to be a compelling way to play Minecraft. Additionally, we discuss the ways that these different types of multimodal data can be used to identify the meaningful spatial reasoning practices that students demonstrate while playing Minecraft. Collectively, this paper emphasizes the opportunity to bridge a multimodal interface with a means for collecting rich data that can better support diverse learners in non-traditional learning environments.