Detecting publication selection bias through excess statistical significance

• Published in: Research synthesis methods Vol. 12; no. 6; pp. 776 - 795
• Main Authors: Stanley, T. D., Doucouliagos, Hristos, Ioannidis, John P. A., Carter, Evan C.
• Format: Journal Article
• Language: English
• Published: Chichester Wiley 01.11.2021; Wiley Subscription Services, Inc
• Subjects: Bias; excess statistical significance; Heterogeneity; Meta-analysis; publication selection bias; Selection; Statistical power; Statistical Significance
• ISSN: 1759-2879; 1759-2887
• DOI: 10.1002/jrsm.1512

We introduce and evaluate three tests for publication selection bias based on excess statistical significance (ESS). The proposed tests incorporate heterogeneity explicitly in the formulas for expected and ESS. We calculate the expected proportion of statistically significant findings in the absence of selective reporting or publication bias based on each study's SE and meta‐analysis estimates of the mean and variance of the true‐effect distribution. A simple proportion of statistical significance test (PSST) compares the expected to the observed proportion of statistically significant findings. Alternatively, we propose a direct test of excess statistical significance (TESS). We also combine these two tests of excess statistical significance (TESSPSST). Simulations show that these ESS tests often outperform the conventional Egger test for publication selection bias and the three‐parameter selection model (3PSM).