Stacked Ensemble Learning for Propensity Score Methods in Observational Studies

• Published in: Journal of educational data mining Vol. 13; no. 1; pp. 24 - 189
• Main Authors: Autenrieth, Maximilian, Levine, Richard A, Fan, Juanjuan, Guarcello, Maureen A
• Format: Journal Article
• Language: English
• Published: International Educational Data Mining 30.06.2021
• Subjects: Artificial Intelligence; College Students; Computation; Educational Research; Observation; Outcomes of Education; Probability; Statistical Bias; California (San Diego)
• DOI: 10.5281/zenodo.5048425
• ISSN: 2157-2100

Propensity score methods account for selection bias in observational studies. However, the consistency of the propensity score estimators strongly depends on a correct specification of the propensity score model. Logistic regression and, with increasing popularity, machine learning tools are used to estimate propensity scores. We introduce a stacked generalization ensemble learning approach to improve propensity score estimation by fitting a meta learner on the predictions of a suitable set of diverse base learners. We perform a comprehensive Monte Carlo simulation study, implementing a broad range of scenarios that mimic characteristics of typical data sets in educational studies. The population average treatment effect is estimated using the propensity score in Inverse Probability of Treatment Weighting. Our proposed stacked ensembles, especially using gradient boosting machines as a meta learner trained on a set of 12 base learner predictions, led to superior reduction of bias compared to the current state-of-the-art in propensity score estimation. Further, our simulations imply that commonly used balance measures (averaged standardized absolute mean differences) might be misleading as propensity score model selection criteria. We apply our proposed model -- which we call GBM-Stack -- to assess the population average treatment effect of a Supplemental Instruction (SI) program in an introductory psychology (PSY 101) course at San Diego State University. Our analysis provides evidence that moving the whole population to SI attendance would on average lead to 1.69 times higher odds to pass the PSY 101 class compared to not offering SI, with a 95% bootstrap confidence interval of (1.31, 2.20).