Pre-Analysis Plans Have Limited Upside, Especially Where Replications Are Feasible

• Published in: The Journal of economic perspectives Vol. 29; no. 3; pp. 81 - 97
• Main Authors: Coffman, Lucas C., Niederle, Muriel
• Format: Journal Article
• Language: English
• Published: Nashville American Economic Association 01.07.2015
• Subjects: Data collection; Datasets; Economic research; Economic theory; Economics; Experimental economics; Experiments; False positive errors; Hypotheses; Laboratories; Logical givens; Political economy; Replicative studies; Research biases; Research papers; Researchers; Social sciences; Statistical significance; Statistics; Studies; Symposium: Pre-Analysis Plans in Economics; United States; United States > US
• ISSN: 0895-3309; 1944-7965
• DOI: 10.1257/jep.29.3.81

The social sciences--including economics--have long called for transparency in research to counter threats to producing robust and replicable results. In this paper, we discuss the pros and cons of three of the more prominent proposed approaches: pre-analysis plans, hypothesis registries, and replications. They have been primarily discussed for experimental research, both in the field including randomized control trials and the laboratory, so we focus on these areas. A pre-analysis plan is a credibly fixed plan of how a researcher will collect and analyze data, which is submitted before a project begins. Though pre-analysis plans have been lauded in the popular press and across the social sciences, we will argue that enthusiasm for pre-analysis plans should be tempered for several reasons. Hypothesis registries are a database of all projects attempted; the goal of this promising mechanism is to alleviate the "file drawer problem," which is that statistically significant results are more likely to be published, while other results are consigned to the researcher's "file drawer." Finally, we evaluate the efficacy of replications. We argue that even with modest amounts of researcher bias--either replication attempts bent on proving or disproving the published work--or modest amounts of poor replication attempts--designs that are underpowered or orthogonal to the hypothesis--replications correct even the most inaccurate beliefs within three to five replications. We offer practical proposals for how to increase the incentives for researchers to carry out replications.