Predicting Dialogue Acts for Intelligent Virtual Agents with Multimodal Student Interaction Data

Recent years have seen a growing interest in intelligent game-based learning environments featuring virtual agents. A key challenge posed by incorporating virtual agents in game-based learning environments is dynamically determining the dialogue moves they should make in order to best support studen...

Full description

Saved in:
Bibliographic Details
Published inInternational Educational Data Mining Society
Main Authors Min, Wookhee, Wiggins, Joseph B, Pezzullo, Lydia G, Vail, Alexandria K, Boyer, Kristy Elizabeth, Mott, Bradford W, Frankosky, Megan H, Wiebe, Eric N, Lester, James C
Format Report
LanguageEnglish
Published International Educational Data Mining Society 2016
Subjects
Accuracy
After School Programs
Computer Games
Computer Simulation
Decision Making
Design
Dialogs (Language)
Intelligent Tutoring Systems
Learning Processes
Microbiology
Middle School Students
Nonverbal Communication
Physiology
Prediction
Problem Solving
Recordkeeping
Role Playing
Sequential Approach
Short Term Memory
Summer Programs
North Carolina (Raleigh)
Online AccessGet full text

Cover

More Information
Summary:Recent years have seen a growing interest in intelligent game-based learning environments featuring virtual agents. A key challenge posed by incorporating virtual agents in game-based learning environments is dynamically determining the dialogue moves they should make in order to best support students' problem solving. This paper presents a data-driven modeling approach that uses a Wizard-of-Oz framework to predict human wizards' dialogue acts based on a sequence of multimodal data streams of student interactions with a game-based learning environment. To effectively deal with multiple, parallel sequential data streams, this paper investigates two sequence-labeling techniques: long short-term memory networks (LSTMs) and conditional random fields. We train predictive models utilizing data corpora collected from two Wizard-of-Oz experiments in which a human wizard played the role of the virtual agent unbeknownst to the student. Empirical results suggest that LSTMs that utilize game trace logs and facial action units achieve the highest predictive accuracy. This work can inform the design of intelligent virtual agents that leverage rich multimodal student interaction data in game-based learning environments.