Multicraft: A Multimodal Interface for Supporting and Studying Learning in Minecraft - Worsley, Marcelo; Mendoza Tudares, Kevin; Mwiti, Timothy; Zhen, Mitchell; Jiang, Marc (HCI in Games: Serious and Immersive Games, 2021)
In this paper, we present work on bringing multimodal interaction to Minecraft. The platform, Multicraft, incorporates speech-based input, eye tracking, and natural language understanding to facilitate more equitable gameplay in Minecraft. We tested the platform with elementary, middle school students and college students through a collection of studies. Students found each of the provided modalities to be a compelling way to play Minecraft. Additionally, we discuss the ways that these different types of multimodal data can be used to identify the meaningful spatial reasoning practices that students demonstrate while playing Minecraft. Collectively, this paper emphasizes the opportunity to bridge a multimodal interface with a means for collecting rich data that can better support diverse learners in non-traditional learning environments.