An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns aro...

Full description

Saved in:
Bibliographic Details
Published inSocial psychological & personality science Vol. 11; no. 4; pp. 464 - 473
Main Authors Chmielewski, Michael, Kucker, Sarah C.
Format Journal Article
LanguageEnglish
Published Los Angeles, CA SAGE Publications 01.05.2020
SAGE PUBLICATIONS, INC
Subjects
Data quality
Personality tests
Quality
Reliability
Research subjects
Social psychology
Summer
Validity
replication
data quality
TurkGate
online samples
Amazon Mechanical Turk
Online AccessGet full text

Cover

More Information
Summary:Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1948-5506
1948-5514
DOI:10.1177/1948550619875149