Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research

• Published in: BMC medical research methodology Vol. 24; no. 1; pp. 108 - 7
• Main Authors: Du, Jingcheng, Soysal, Ekin, Wang, Dong, He, Long, Lin, Bin, Wang, Jingqi, Manion, Frank J., Li, Yeran, Wu, Elise, Yao, Lixia
• Format: Journal Article
• Language: English
• Published: London BioMed Central 09.05.2024; BioMed Central Ltd; BMC
• Subjects: Abstracting and Indexing - methods; Algorithms; Application programming interface; Article screening; Automation; Bibliographic data bases; Classification; Cost control; Deep Learning; Economics, Medical; Evaluation; Health economics; Health Sciences; Human papillomavirus; Humans; Keywords; Literature reviews; Machine Learning; Medical economics; Medical research; Medical screening; Medicine; Medicine & Public Health; Medicine, Experimental; Metadata; Methods; Natural language; Outcome Assessment, Health Care - methods; Papillomavirus Infections - diagnosis; Pediatrics; R&D; Regression analysis; Research & development; Statistical Theory and Methods; Statistics for Life Sciences; Subject specialists; Support vector machines; Systematic literature review; Text categorization; Text classification; Theory of Medicine/Bioethics; Treatment outcome; United States; Deep learning; Article screening; Machine learning; Text classification; Systematic literature review
• ISSN: 1471-2288; 1471-2288
• DOI: 10.1186/s12874-024-02224-3

Objective Systematic literature reviews (SLRs) are critical for life-science research. However, the manual selection and retrieval of relevant publications can be a time-consuming process. This study aims to (1) develop two disease-specific annotated corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases (PAPD), and (2) optimize machine- and deep-learning models to facilitate automation of the SLR abstract screening. Methods This study constructed two disease-specific SLR screening corpora for HPV and PAPD, which contained citation metadata and corresponding abstracts. Performance was evaluated using precision, recall, accuracy, and F1-score of multiple combinations of machine- and deep-learning algorithms and features such as keywords and MeSH terms. Results and conclusions The HPV corpus contained 1697 entries, with 538 relevant and 1159 irrelevant articles. The PAPD corpus included 2865 entries, with 711 relevant and 2154 irrelevant articles. Adding additional features beyond title and abstract improved the performance (measured in Accuracy) of machine learning models by 3% for HPV corpus and 2% for PAPD corpus. Transformer-based deep learning models that consistently outperformed conventional machine learning algorithms, highlighting the strength of domain-specific pre-trained language models for SLR abstract screening. This study provides a foundation for the development of more intelligent SLR systems.