Retrospective median power, false positive meta‐analysis and large‐scale replication

• Published in: Research synthesis methods Vol. 13; no. 1; pp. 88 - 108
• Main Authors: Stanley, T. D., Doucouliagos, Hristos, Ioannidis, John P. A.
• Format: Journal Article
• Language: English
• Published: England Wiley 01.01.2022; Wiley Subscription Services, Inc
• Subjects: Credibility; Educational research; false positives; Median (statistics); Meta Analysis; power; Publication Bias; Replication (Evaluation); replications; Research methodology; Research Problems; Retrospective Studies; Statistical Analysis; meta-analysis; credibility; replications; power; false positives
• ISSN: 1759-2879; 1759-2887
• DOI: 10.1002/jrsm.1529

Recent, high‐profile, large‐scale, preregistered failures to replicate uncover that many highly‐regarded experiments are “false positives”; that is, statistically significant results of underlying null effects. Large surveys of research reveal that statistical power is often low and inadequate. When the research record includes selective reporting, publication bias and/or questionable research practices, conventional meta‐analyses are also likely to be falsely positive. At the core of research credibility lies the relation of statistical power to the rate of false positives. This study finds that high (>50%–60%) median retrospective power (MRP) is associated with credible meta‐analysis and large‐scale, preregistered, multi‐lab “successful” replications; that is, with replications that corroborate the effect in question. When median retrospective power is low (<50%), positive meta‐analysis findings should be interpreted with great caution or discounted altogether.