Developing Standards for Post-Hoc Weighting in Population-Based Survey Experiments

Weighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selecti...

Full description

Saved in:
Bibliographic Details
Published inJournal of experimental political science Vol. 4; no. 2; pp. 161 - 172
Main Authors Franco, Annie, Malhotra, Neil, Simonovits, Gabor, Zigerell, L. J.
Format Journal Article
LanguageEnglish
Published New York, USA Cambridge University Press 2017
Subjects
Bias
Experiments
Heterogeneity
Political science
Polls & surveys
Public opinion surveys
Research methodology
Transparency
Validity
Weighting
Survey experiment
representativeness
transparency
PATE
SATE
weighting
external validity
Online AccessGet full text

Cover

More Information
Summary:Weighting techniques are employed to generalize results from survey experiments to populations of theoretical and substantive interest. Although weighting is often viewed as a second-order methodological issue, these adjustment methods invoke untestable assumptions about the nature of sample selection and potential heterogeneity in the treatment effect. Therefore, although weighting is a useful technique in estimating population quantities, it can introduce bias and also be used as a researcher degree of freedom. We review survey experiments published in three major journals from 2000–2015 and find that there are no standard operating procedures for weighting survey experiments. We argue that all survey experiments should report the sample average treatment effect (SATE). Researchers seeking to generalize to a broader population can weight to estimate the population average treatment effect (PATE), but should discuss the construction and application of weights in a detailed and transparent manner given the possibility that weighting can introduce bias.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2052-2630
2052-2649
DOI:10.1017/XPS.2017.2