Evaluating and improving adaptive educational systems with learning curves

• Published in: User modeling and user-adapted interaction Vol. 21; no. 3; pp. 249 - 283
• Main Authors: Martin, Brent, Mitrovic, Antonija, Koedinger, Kenneth R., Mathan, Santosh
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01.08.2011; Springer Nature B.V
• Subjects: Adaptive systems; Computer Science; Degradation; Design engineering; Education; Educational materials; Experiments; Feasibility; Feedback; Knowledge; Learning; Learning curves; Management of Computing and Information Systems; Mathematical models; Multimedia Information Systems; Original Paper; Pedagogy; Performance enhancement; Students; Studies; Tutoring; User Interfaces and Human Computer Interaction; Users; United States > US; Learning curves; Domain modelling; Student modeling; User modelling; Empirical evaluation; Intelligent tutoring systems
• ISSN: 0924-1868; 1573-1391
• DOI: 10.1007/s11257-010-9084-2

Personalised environments such as adaptive educational systems can be evaluated and compared using performance curves. Such summative studies are useful for determining whether or not new modifications enhance or degrade performance. Performance curves also have the potential to be utilised in formative studies that can shape adaptive model design at a much finer level of granularity. We describe the use of learning curves for evaluating personalised educational systems and outline some of the potential pitfalls and how they may be overcome. We then describe three studies in which we demonstrate how learning curves can be used to drive changes in the user model. First, we show how using learning curves for subsets of the domain model can yield insight into the appropriateness of the model’s structure. In the second study we use this method to experiment with model granularity. Finally, we use learning curves to analyse a large volume of user data to explore the feasibility of using them as a reliable method for fine-tuning a system’s model. The results of these experiments demonstrate the successful use of performance curves in formative studies of adaptive educational systems.