AutoML Feature Engineering for Student Modeling Yields High Accuracy, but Limited Interpretability

Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational...

Full description

Saved in:
Bibliographic Details
Published inJournal of educational data mining Vol. 13; no. 2; pp. 55 - 79
Main Author Bosch, Nigel
Format Journal Article
LanguageEnglish
Published International Educational Data Mining 2021
Subjects
Accuracy
Comparative Analysis
Competition
Computer Software
Data Analysis
Decision Making
Evaluators
Grade 4
Grade 8
Hypothesis Testing
Learning Analytics
Mathematics Tests
National Competency Tests
Prediction
Specialists
Time Management
Online AccessGet full text

Cover

More Information
Summary:Automatic machine learning (AutoML) methods automate the time-consuming, feature-engineering process so that researchers produce accurate student models more quickly and easily. In this paper, we compare two AutoML feature engineering methods in the context of the National Assessment of Educational Progress (NAEP) data mining competition. The methods we compare, Featuretools and TSFRESH (Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests), have rarely been applied in the context of student interaction log data. Thus, we address research questions regarding the accuracy of models built with AutoML features, how AutoML feature types compare to each other and to expert-engineered features, and how interpretable the features are. Additionally, we developed a novel feature selection method that addresses problems applying AutoML feature engineering in this context, where there were many heterogeneous features (over 4,000) and relatively few students. Our entry to the NAEP competition placed 3rd overall on the final held-out dataset and 1st on the public leaderboard, with a final Cohen's kappa = 0.212 and area under the receiver operating characteristic curve (AUC) = 0.665 when predicting whether students would manage their time effectively on a math assessment. We found that TSFRESH features were significantly more effective than either Featuretools features or expert-engineered features in this context; however, they were also among the most difficult features to interpret based on a survey of six experts' judgments. Finally, we discuss the tradeoffs between effort and interpretability that arise in AutoML-based student modeling.
ISSN:2157-2100
2157-2100