Increasing Precision without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments

• Published in: The American political science review Vol. 115; no. 3; pp. 1048 - 1065
• Main Authors: CLIFFORD, SCOTT, SHEAGLEY, GEOFFREY, PISTON, SPENCER
• Format: Journal Article
• Language: English
• Published: New York, USA Cambridge University Press 01.08.2021
• Subjects: Between-subjects design; Collaboration; Dependent variables; Estimates; Evidence; Experiments; Fear & phobias; Heterogeneity; Inferences; Meta Analysis; Outcome Measures; Political science; Polls & surveys; Public Opinion; Recruitment; Researchers; Resistance (Psychology); Scientists; Statistical power; Wisdom; Within-subjects design
• ISSN: 0003-0554; 1537-5943
• DOI: 10.1017/S0003055421000241

The use of survey experiments has surged in political science. The most common design is the between-subjects design in which the outcome is only measured posttreatment. This design relies heavily on recruiting a large number of subjects to precisely estimate treatment effects. Alternative designs that involve repeated measurements of the dependent variable promise greater precision, but they are rarely used out of fears that these designs will yield different results than a standard design (e.g., due to consistency pressures). Across six studies, we assess this conventional wisdom by testing experimental designs against each other. Contrary to common fears, repeated measures designs tend to yield the same results as more common designs while substantially increasing precision. These designs also offer new insights into treatment effect size and heterogeneity. We conclude by encouraging researchers to adopt repeated measures designs and providing guidelines for when and how to use them.