The online laboratory: conducting experiments in a real labor market

• Published in: Experimental economics : a journal of the Economic Science Association Vol. 14; no. 3; pp. 399 - 425
• Main Authors: Horton, John J., Rand, David G., Zeckhauser, Richard J.
• Format: Journal Article
• Language: English
• Published: Boston Springer US 01.09.2011; Springer Science + Business Media B.V; Springer; Springer Nature B.V
• Series: Experimental Economics
• Subjects: Application programming interface; Arbeitsmarkt; Behavioral/Experimental Economics; C70; C91; C92; C93; Computer terminals; Cooperation; Datenbank; Design; Economic theory; Economic Theory/Quantitative Economics/Mathematical Methods; Economics; Economics and Finance; Experimentation; Experiments; Field experiment; Field work; Game Theory; Interfaces; Internet; Internetbasierter Dienst; Labor economics; Labor market; Labor supply; Laboratories; Labour economics; Labour market; Labour supply; Microeconomics; Online labor markets; Operations Research/Decision Theory; Prisoner's dilemma; Social and Behav. Sciences; Software; Software development; Studies; Test; Validity; Workshops; Prisoner’s dilemma; J2; Experimentation; C91; Internet; C93; C92; C70; Field experiment; Online labor markets
• ISSN: 1386-4157; 1573-6938
• DOI: 10.1007/s10683-011-9273-9

Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context. Online experiments, we show, can be just as valid—both internally and externally—as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments. The first finds quantitative agreement between levels of cooperation in a prisoner’s dilemma played online and in the physical laboratory. The second shows—consistent with behavior in the traditional laboratory—that online subjects respond to priming by altering their choices. The third demonstrates that when an identical decision is framed differently, individuals reverse their choice, thus replicating a famed Tversky-Kahneman result. Then we conduct a field experiment showing that workers have upward-sloping labor supply curves. Finally, we analyze the challenges to online experiments, proposing methods to cope with the unique threats to validity in an online setting, and examining the conceptual issues surrounding the external validity of online results. We conclude by presenting our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.