System-, teacher-, and student-level interventions for improving participation in online learning at scale in high schools

• Published in: Proceedings of the National Academy of Sciences - PNAS Vol. 120; no. 30; p. e2216686120
• Main Authors: Asanov, Igor, Asanov, Anastasiya-Mariya, Åstebro, Thomas, Buenstorf, Guido, Crépon, Bruno, McKenzie, David, Flores T., Francisco Pablo, Mensmann, Mona, Schulte, Mathis
• Format: Journal Article
• Language: English
• Published: United States National Academy of Sciences 25.07.2023
• Subjects: CAI; Computer assisted instruction; COVID-19; Distance learning; Incentives; Internet; Learning; Messages; Monitoring; Online instruction; Pandemics; Schools; Social Sciences; Students; Teachers; nudge interventions; scale experiments; online education; centralized monitoring
• ISSN: 0027-8424; 1091-6490; 1091-6490
• DOI: 10.1073/pnas.2216686120

Many school systems across the globe turned to online education during the COVID-19 pandemic. This context differs significantly from the prepandemic situation in which massive open online courses attracted large numbers of voluntary learners who struggled with completion. Students who are provided online courses by their high schools also have their behavior determined by actions of their teachers and school system. We conducted experiments to improve participation in online learning before, during, and right after the COVID-19 outbreak, with 1,151 schools covering more than 45,000 students in their final years of high school in Ecuador. These experiments tested light-touch interventions at scale, motivated by behavioral science, and were carried out at three levels: that of the system, teacher, and student. We find the largest impacts come from intervening at the system level. A cheap, online learning management system for centralized monitoring increased participation by 0.21 SD and subject knowledge by 0.13 SD relative to decentralized management. Centralized management is particularly effective for underperforming schools. Teacher-level nudges in the form of benchmarking emails, encouragement messages, and administrative reminders did not improve student participation. There was no significant impact of encouragement messages to students, or in having them plan and team-up with peers. Small financial incentives in the form of lottery prizes for finishing lessons did increase study time, but was less cost-effective, and had no significant impact on knowledge. The results show the difficulty in incentivizing online learning at scale, and a key role for central monitoring.