Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers

• Published in: Grantee Submission Vol. 11; no. 2; pp. 267 - 295
• Main Author: Porter, Kristin E.
• Format: Journal Article Web Resource
• Language: English
• Published: Philadelphia Taylor & Francis Ltd 03.04.2018
• Subjects: Computation; Correlation; Educational Research; Effect Size; Evaluation Methods; Hypothesis Testing; Intervention; Monte Carlo Methods; Program Effectiveness; Researchers; Sample Size; Statistical Analysis; Statistical power
• ISSN: 1934-5747; 1934-5739
• DOI: 10.1080/19345747.2017.1342887

Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs.