The use of propensity scores to assess the generalizability of results from randomized trials

• Published in: Journal of the Royal Statistical Society. Series A, Statistics in society Vol. 174; no. 2; pp. 369 - 386
• Main Authors: Stuart, Elizabeth A., Cole, Stephen R., Bradshaw, Catherine P., Leaf, Philip J.
• Format: Journal Article
• Language: English
• Published: Oxford, UK Blackwell Publishing Ltd 01.04.2011; Blackwell Publishing; Blackwell; Royal Statistical Society; Oxford University Press
• Series: Journal of the Royal Statistical Society Series A
• Subjects: Applications; Causal inference; Combinatorics; Combinatorics. Ordered structures; Control groups; Designs and configurations; Effects; Elementary schools; Empirical research; Estimating techniques; Estimation; Exact sciences and technology; Experimental design; Experimentation; Experiments; External validity; General topics; Generalization; Inference; Intervention; Mathematics; Methodology; Observational studies; Outcomes of education; Population estimates; Population mean; Positive behavioral interventions and supports; Probability and statistics; Random sampling; Research synthesis; Sampling methods; Sciences and techniques of general use; State schools; Statistical analysis; Statistical methods; Statistics; Valuation; Statistical method; Prevention; Causal inference; Positive behavioral interventions and supports; Randomized design; Experimental design; Research synthesis; External validity; Statistical estimation; Covariate; Implementation; Positive Behavioral Interventions and Supports
• ISSN: 0964-1998; 1467-985X
• DOI: 10.1111/j.1467-985X.2010.00673.x

Randomized trials remain the most accepted design for estimating the effects of interventions, but they do not necessarily answer a question of primary interest: will the programme be effective in a target population in which it may be implemented? In other words, are the results generalizable? There has been very little statistical research on how to assess the generalizability, or `external validity', of randomized trials. We propose the use of propensity-score-based metrics to quantify the similarity of the participants in a randomized trial and a target population. In this setting the propensity score model predicts participation in the randomized trial, given a set of covariates. The resulting propensity scores are used first to quantify the difference between the trial participants and the target population, and then to match, subclassify or weight the control group outcomes to the population, assessing how well the propensity-score-adjusted outcomes track the outcomes that are actually observed in the population. These metrics can serve as a first step in assessing the generalizability of results from randomized trials to target populations. The paper lays out these ideas, discusses the assumptions underlying the approach and illustrates the metrics by using data on the evaluation of a schoolwide prevention programme called `Positive behavioral interventions and supports'.