Robustness of evidence reported in preprints during peer review

• Published in: The Lancet global health Vol. 10; no. 11; pp. e1684 - e1687
• Main Authors: Nelson, Lindsay, Ye, Honghan, Schwenn, Anna, Lee, Shinhyo, Arabi, Salsabil, Hutchins, B Ian
• Format: Journal Article
• Language: English
• Published: England Elsevier Ltd 01.11.2022; The Author(s). Published by Elsevier Ltd
• Subjects: Biomedical Research; COVID-19 - epidemiology; Disease Outbreaks; Health Policy; Humans; Pandemics; Peer Review
• ISSN: 2214-109X; 2214-109X
• DOI: 10.1016/S2214-109X(22)00368-0

Scientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Expert peer-review scores of preprint quality were not related to eventual publication in a peer-reviewed journal. Uncertainty was reduced during peer review, with CIs reducing by 7% on average. These results support the use of preprints, a component of biomedical research literature, in decision making. These results can also help inform the use of preprints during the ongoing COVID-19 pandemic and future disease outbreaks.