Conjuring Power from a Theory of Change: The PWRD Method for Trials with Anticipated Variation in Effects

• Published in: Journal of research on educational effectiveness Vol. 16; no. 4; pp. 707 - 733
• Main Authors: Lycurgus, Timothy, Hansen, Ben B., White, Mark
• Format: Journal Article
• Language: English
• Published: Philadelphia Routledge 02.10.2023; Taylor & Francis Ltd
• Subjects: At Risk Students; Educational research; Educational tests & measurements; Educational theory; Elementary School Students; Enhanced statistical precision; logic model; Randomized Controlled Trials; Reading Difficulties; repeated measures design; varying length of follow-up; Within-subjects design
• ISSN: 1934-5747; 1934-5739
• DOI: 10.1080/19345747.2022.2142178

We present an aggregation scheme that increases power in randomized controlled trials and quasi-experiments when the intervention possesses a robust and well-articulated theory of change. Intervention studies using longitudinal data often include multiple observations on individuals, some of which may be more likely to manifest a treatment effect than others. An intervention's theory of change provides guidance as to which of those observations are best situated to exhibit that treatment effect. Our power-maximizing weighting for repeated-measurements with delayed-effects scheme, PWRD aggregation, converts the theory of change into a test statistic with improved asymptotic relative efficiency, delivering tests with greater statistical power. We illustrate this method on an IES-funded cluster randomized trial testing the efficacy of a reading intervention designed to assist early elementary students at risk of falling behind their peers. The salient theory of change holds program benefits to be delayed and non-uniform, experienced after a student's performance stalls. In this instance, the PWRD technique's effect on power is found to be comparable to that of doubling the number of clusters in the experiment.