Benefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor

• Published in: Speech communication Vol. 53; no. 9; pp. 1115 - 1136
• Main Authors: Forbes-Riley, Kate, Litman, Diane
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2011
• Subjects: Accompanying manual annotation; Adaptation; Adaptive systems; Automatic affect detection and adaptation; Controlled experimental evaluation; Dynamical systems; Dynamics; Emotions; Error Analysis; Fully automated spoken dialogue tutoring system; Human Computer Communication; Machine Learning; Speech; Speech Synthesis; Students; Uncertainty; Fully automated spoken dialogue tutoring system; Automatic affect detection and adaptation; Accompanying manual annotation; Error analysis; Controlled experimental evaluation
• ISSN: 0167-6393; 1872-7182
• DOI: 10.1016/j.specom.2011.02.006

We evaluate the performance of a spoken dialogue system that provides substantive dynamic responses to automatically detected user affective states. We then present a detailed system error analysis that reveals challenges for real-time affect detection and adaptation. This research is situated in the tutoring domain, where the user is a student and the spoken dialogue system is a tutor. Our adaptive system detects uncertainty in each student turn via a model that combines a machine learning approach with hedging phrase heuristics; the learned model uses acoustic-prosodic and lexical features extracted from the speech signal, as well as dialogue features. The adaptive system varies its content based on the automatic uncertainty and correctness labels for each turn. Our controlled experimental evaluation shows that the adaptive system yields higher global performance than two non-adaptive control systems, but the difference is only significant for a subset of students. Our system error analysis indicates that noisy affect labeling is a major performance bottleneck, yielding fewer than expected adaptations thus lower than expected performance. However, the percentage of received adaptation correlates with higher performance over all students. Moreover, when uncertainty is accurately recognized and adapted to, local performance is significantly improved.