The Generalizability of Online Experiments Conducted During the COVID-19 Pandemic

The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the compositi...

Full description

Saved in:
Bibliographic Details
Published inJournal of experimental political science Vol. 9; no. 3; pp. 379 - 394
Main Authors Peyton, Kyle, Huber, Gregory A., Coppock, Alexander
Format Journal Article
LanguageEnglish
Published New York, USA Cambridge University Press 01.01.2022
Subjects
Coronaviruses
COVID-19
Estimates
Experiments
Internet
Pandemics
Political science
Polls & surveys
Research Article
Research methodology
Social scientists
Time periods
Validity
United States > US
generalizability
replication
COVID-19
experiment
attentiveness
survey research
pandemic
online research
Online AccessGet full text

Cover

More Information
Summary:The COVID-19 pandemic imposed new constraints on empirical research, and online data collection by social scientists increased. Generalizing from experiments conducted during this period of persistent crisis may be challenging due to changes in how participants respond to treatments or the composition of online samples. We investigate the generalizability of COVID era survey experiments with 33 replications of 12 pre-pandemic designs, fielded across 13 quota samples of Americans between March and July 2020. We find strong evidence that pre-pandemic experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. Indirect evidence suggests an increased share of inattentive subjects on online platforms during this period, which may have contributed to smaller estimated treatment effects. Overall, we conclude that the pandemic does not pose a fundamental threat to the generalizability of online experiments to other time periods.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2052-2630
2052-2649
DOI:10.1017/XPS.2021.17