The Mechanics of Treatment-effect Estimate Bias for Nonexperimental Data

• Published in: Sociological methods & research Vol. 51; no. 1; pp. 165 - 202
• Main Authors: Penaloza, Roberto V., Berends, Mark
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.02.2022; SAGE PUBLICATIONS, INC
• Subjects: Bias; Correlation; Data; Error of Measurement; Measurement; Measurement errors; Simulation; Social science education; Statistical Analysis; Statistical Bias; Student teacher relationship; Teacher education; Teachers; Value Added Models; treatment-effect estimate bias; measurement error; school effectiveness; value-added models; teacher effectiveness
• ISSN: 0049-1241; 1552-8294
• DOI: 10.1177/0049124119852375

To measure “treatment” effects, social science researchers typically rely on nonexperimental data. In education, school and teacher effects on students are often measured through value-added models (VAMs) that are not fully understood. We propose a framework that relates to the education production function in its most flexible form and connects with the basic VAMs without using untenable assumptions. We illustrate how, due to measurement error (ME), cross-group imbalances created by nonrandom group assignment cause correlations that drive the models’ treatment-effect estimate bias. We derive formulas to calculate bias and rank the models and show that no model is better in all situations. The framework and formulas’ workings are verified and illustrated via simulation. We also evaluate the performance of latent variable/errors-in-variables models that handle ME and study the role of extra covariates including lags of the outcome.