A Bayesian bird's eye view of ‘Replications of important results in social psychology’

• Published in: Royal Society open science Vol. 4; no. 1; p. 160426
• Main Authors: Marsman, Maarten, Schönbrodt, Felix D., Morey, Richard D., Yao, Yuling, Gelman, Andrew, Wagenmakers, Eric-Jan
• Format: Journal Article
• Language: English
• Published: England The Royal Society Publishing 01.01.2017; The Royal Society
• Subjects: Bayes Factor; Credible Interval; Evidence; Preregistration; Psychology And Cognitive Neuroscience; Reproducibility; Bayes factor; preregistration; evidence; reproducibility; credible interval
• ISSN: 2054-5703; 2054-5703
• DOI: 10.1098/rsos.160426

We applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue ‘Replications of Important Results in Social Psychology’ (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141. (doi:10.1027/1864-9335/a000192)). First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only three out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only four out of 59 credible intervals contained values greater than 0.10 (10% of variance explained) and only 19 intervals contained values larger than 0.05. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only seven out of 60 Bayes factors indicated non-anecdotal support in favour of the alternative hypothesis (BF10>3), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques.