Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial

• Published in: Journal of clinical epidemiology Vol. 121; pp. 20 - 28
• Main Authors: Gartlehner, Gerald, Affengruber, Lisa, Titscher, Viktoria, Noel-Storr, Anna, Dooley, Gordon, Ballarini, Nicolas, König, Franz
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.05.2020; Elsevier Limited
• Subjects: Abstracting and Indexing - standards; Abstracting and Indexing - statistics & numerical data; Accuracy; Adult; Bibliographic data bases; Clinical trials; Confidence intervals; Data Accuracy; Decision making; Depression - therapy; Epidemiology; Female; Humans; Internal Medicine; Literature reviews; Literature screening; Male; Peer Review, Research - standards; Public health; Quality standards; Random Allocation; Randomization; Randomized controlled trial; Rapid reviews; Regression Analysis; Sample Size; Screening; Sensitivity; Sensitivity and Specificity; Skills; Standard deviation; Studies; Sugar-Sweetened Beverages; Systematic review; Systematic reviews; Systematic Reviews as Topic; United States > US; Literature screening; Systematic reviews; Rapid reviews; Accuracy; Randomized controlled trial; rapid reviews; accuracy; systematic reviews; literature screening
• DOI: 10.1016/j.jclinepi.2020.01.005
• ISSN: 0895-4356; 1878-5921

To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews. We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards. Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%–91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%–98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%–80.9%) and 68.7% (95% CI, 66.4%–71.0%), respectively. Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.