On the Transportability of Laboratory Results

• Published in: Sociological methods & research Vol. 50; no. 3; pp. 1452 - 1481
• Main Authors: Bader, Felix, Baumeister, Bastian, Berger, Roger, Keuschnigg, Marc
• Format: Journal Article
• Language: English
• Published: Los Angeles, CA SAGE Publications 01.08.2021; SAGE PUBLICATIONS, INC
• Subjects: Anonymity; Behavior; Change agents; College Students; Experimental methods; external validity; Foreign Countries; Games; Generalizability; Generalization; Incentives; Laboratories; Laboratory Experiments; laboratory research; Measurement; mode effects; online experiments; Prosocial Behavior; Reciprocity; Replication (Evaluation); Robustness; sample effects; Sampling; Social Science Research; Validity; Germany; online experiments; prosocial behavior; sample effects; mode effects; anonymity; laboratory research; external validity; experimental methods
• ISSN: 0049-1241; 1552-8294; 1552-8294
• DOI: 10.1177/0049124119826151

The “transportability” of laboratory findings to other instances than the original implementation entails the robustness of rates of observed behaviors and estimated treatment effects to changes in the specific research setting and in the sample under study. In four studies based on incentivized games of fairness, trust, and reciprocity, we evaluate (1) the sensitivity of laboratory results to locally recruited student-subject pools, (2) the comparability of behavioral data collected online and, under varying anonymity conditions, in the laboratory, (3) the generalizability of student-based results to the broader population, and (4) with a replication at Amazon Mechanical Turk, the stability of laboratory results across research contexts. For the class of laboratory designs using incentivized games as measurement instruments of prosocial behavior, we find that rates of behavior and the exact behavioral differences between decision situations do not transport beyond specific implementations. Most clearly, data obtained from standard participant pools differ significantly from those from the broader population. This undermines the use of empirically motivated laboratory studies to establish descriptive parameters of human behavior. Directions of the behavioral differences between games, in contrast, are remarkably robust to changes in samples and settings. Moreover, we find no evidence for either anonymity effects nor mode effects potentially biasing laboratory measurement. These results underscore the capacity of laboratory experiments to establish generalizable causal effects in theory-driven designs.