A new algorithm for reducing the workload of experts in performing systematic reviews

• Published in: Journal of the American Medical Informatics Association : JAMIA Vol. 17; no. 4; pp. 446 - 453
• Main Authors: Matwin, Stan, Kouznetsov, Alexandre, Inkpen, Diana, Frunza, Oana, O'Blenis, Peter
• Format: Journal Article
• Language: English
• Published: England BMJ Group 01.07.2010
• Subjects: Algorithms; Automation; Bayes Theorem; Decision Support Techniques; Drug Therapy; Efficiency; Evidence-Based Medicine - classification; Humans; Information Storage and Retrieval - classification; Research Paper; Review Literature as Topic
• ISSN: 1067-5027; 1527-974X; 1527-974X
• DOI: 10.1136/jamia.2010.004325

To determine whether a factorized version of the complement naïve Bayes (FCNB) classifier can reduce the time spent by experts reviewing journal articles for inclusion in systematic reviews of drug class efficacy for disease treatment. The proposed classifier was evaluated on a test collection built from 15 systematic drug class reviews used in previous work. The FCNB classifier was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Weight engineering (WE) techniques were added to reduce underestimation for Medical Subject Headings (MeSH)-based and Publication Type (PubType)-based features. Cross-validation experiments were performed to evaluate the classifier's parameters and performance. Work saved over sampling (WSS) at no less than a 95% recall was used as the main measure of performance. The minimum workload reduction for a systematic review for one topic, achieved with a FCNB/WE classifier, was 8.5%; the maximum was 62.2% and the average over the 15 topics was 33.5%. This is 15.0% higher than the average workload reduction obtained using a voting perceptron-based automated citation classification system. The FCNB/WE classifier is simple, easy to implement, and produces significantly better results in reducing the workload than previously achieved. The results support it being a useful algorithm for machine-learning-based automation of systematic reviews of drug class efficacy for disease treatment.