Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants

• Published in: PloS one Vol. 11; no. 6; p. e0157732
• Main Authors: Necka, Elizabeth A., Cacioppo, Stephanie, Norman, Greg J., Cacioppo, John T.
• Format: Journal Article
• Language: English
• Published: United States Public Library of Science 28.06.2016; Public Library of Science (PLoS)
• Subjects: Adult; Aged; Aged, 80 and over; Behavior; Bias; Biology and Life Sciences; Crowdsourcing; Data Accuracy; Female; Humans; Influence; Laboratories; Male; Middle Aged; People and Places; Prevalence studies (Epidemiology); Problem Behavior; Quality; Quality assessment; Research and Analysis Methods; Research Subjects - psychology; Research Subjects - statistics & numerical data; Researchers; Sample Size; Samples; Social Sciences; Statistical analysis; Statistical methods; Statistics; Students - psychology; Students - statistics & numerical data; Studies; Surveys and Questionnaires - standards; Universities - statistics & numerical data; United States; United States > US; Illinois; Chicago Illinois
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0157732

The reliance on small samples and underpowered studies may undermine the replicability of scientific findings. Large sample sizes may be necessary to achieve adequate statistical power. Crowdsourcing sites such as Amazon's Mechanical Turk (MTurk) have been regarded as an economical means for achieving larger samples. Because MTurk participants may engage in behaviors which adversely affect data quality, much recent research has focused on assessing the quality of data obtained from MTurk samples. However, participants from traditional campus- and community-based samples may also engage in behaviors which adversely affect the quality of the data that they provide. We compare an MTurk, campus, and community sample to measure how frequently participants report engaging in problematic respondent behaviors. We report evidence that suggests that participants from all samples engage in problematic respondent behaviors with comparable rates. Because statistical power is influenced by factors beyond sample size, including data integrity, methodological controls must be refined to better identify and diminish the frequency of participant engagement in problematic respondent behaviors.