Meta-analyses in psychology often overestimate evidence for and size of effects

• Published in: Royal Society open science Vol. 10; no. 7; p. 230224
• Main Authors: Bartoš, František, Maier, Maximilian, Shanks, David R, Stanley, T D, Sladekova, Martina, Wagenmakers, Eric-Jan
• Format: Journal Article
• Language: English
• Published: England The Royal Society 05.07.2023
• Subjects: Bayesian inference; meta-analysis; model-averaging; model-selection; Psychology and Cognitive Neuroscience; publication bias; RoBMA; model-selection; meta-analysis; publication bias; model-averaging; Bayesian inference; RoBMA
• ISSN: 2054-5703; 2054-5703
• DOI: 10.1098/rsos.230224

Adjusting for publication bias is essential when drawing meta-analytic inferences. However, most methods that adjust for publication bias do not perform well across a range of research conditions, such as the degree of heterogeneity in effect sizes across studies. Sladekova . 2022 (Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods. ) tried to circumvent this complication by selecting the methods that are most appropriate for a given set of conditions, and concluded that publication bias on average causes only minimal over-estimation of effect sizes in psychology. However, this approach suffers from a 'Catch-22' problem-to know the underlying research conditions, one needs to have adjusted for publication bias correctly, but to correctly adjust for publication bias, one needs to know the underlying research conditions. To alleviate this problem, we conduct an alternative analysis, robust Bayesian meta-analysis (RoBMA), which is not based on but on . In RoBMA, models that predict the observed results better are given correspondingly larger weights. A RoBMA reanalysis of Sladekova .'s dataset reveals that more than 60% of meta-analyses in psychology notably overestimate the evidence for the presence of the meta-analytic effect and more than 50% overestimate its magnitude.