Predicting student outcomes using digital logs of learning behaviors: Review, current standards, and suggestions for future work

• Published in: Behavior research methods Vol. 55; no. 6; pp. 3026 - 3054
• Main Authors: Arizmendi, Cara J., Bernacki, Matthew L., Raković, Mladen, Plumley, Robert D., Urban, Christopher J., Panter, A. T., Greene, Jeffrey A., Gates, Kathleen M.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.09.2023; Springer Nature B.V
• Subjects: Behavioral Science and Psychology; Cognitive Psychology; Education; Learning management systems; Literature reviews; Psychology; Well being; Equity; Data privacy; Learning management system; Machine learning; Digital data
• ISSN: 1554-3528; 1554-351X; 1554-3528
• DOI: 10.3758/s13428-022-01939-9

Using traces of behaviors to predict outcomes is useful in varied contexts ranging from buyer behaviors to behaviors collected from smart-home devices. Increasingly, higher education systems have been using Learning Management System (LMS) digital data to capture and understand students’ learning and well-being. Researchers in the social sciences are increasingly interested in the potential of using digital log data to predict outcomes and design interventions. Using LMS data for predicting the likelihood of students’ success in for-credit college courses provides a useful example of how social scientists can use these techniques on a variety of data types. Here, we provide a primer on how LMS data can be feature-mapped and analyzed to accomplish these goals. We begin with a literature review summarizing current approaches to analyzing LMS data, then discuss ethical issues of privacy when using demographic data and equitable model building. In the second part of the paper, we provide an overview of popular machine learning algorithms and review analytic considerations such as feature generation, assessment of model performance, and sampling techniques. Finally, we conclude with an empirical example demonstrating the ability of LMS data to predict student success, summarizing important features and assessing model performance across different model specifications.