Combining a Local Comparison Group, a Pretest Measure, and Rich Covariates: How Well Do They Collectively Reduce Bias in Nonequivalent Comparison Group Designs?

This study examined bias reduction in the eight nonequivalent comparison group designs (NECGDs) that result from combining (a) choice of a local versus non-local comparison group, and analytic use or not of (b) a pretest measure of the study outcome and (c) a rich set of other covariates. Bias was e...

Full description

Saved in:
Bibliographic Details
Published inAmerican educational research journal Vol. 60; no. 1; pp. 141 - 182
Main Authors Brown, Seth, Song, Mengli, Cook, Thomas D., Garet, Michael S.
Format Journal Article
LanguageEnglish
Published Los Angeles, CA SAGE Publications 01.02.2023
American Educational Research Association
Subjects
Bias
Computation
Control Groups
Educational Research
Intervention
Pretests Posttests
Program Effectiveness
Randomized Controlled Trials
Replication (Evaluation)
Research Design
within-study comparison
selection bias reduction
design features
Online AccessGet full text

Cover

More Information
Summary:This study examined bias reduction in the eight nonequivalent comparison group designs (NECGDs) that result from combining (a) choice of a local versus non-local comparison group, and analytic use or not of (b) a pretest measure of the study outcome and (c) a rich set of other covariates. Bias was estimated as the difference in causal estimate between each NECGD and a carefully appraised randomized experiment with the same intervention, outcome, and estimand. Results indicated that bias generally declined with the number of design elements in an NECGD, that combining all three sufficed to eliminate bias but was not necessary for it, and that this pattern of results was largely replicated across five different replication factors.
ISSN:0002-8312
1935-1011
DOI:10.3102/00028312221136565