Validation of an Electronic Health Record–Based Machine Learning Model Compared With Clinical Risk Scores for Gastrointestinal Bleeding

• Published in: Gastroenterology (New York, N.Y. 1943) Vol. 167; no. 6; pp. 1198 - 1212
• Main Authors: Shung, Dennis L., Chan, Colleen E., You, Kisung, Nakamura, Shinpei, Saarinen, Theo, Zheng, Neil S., Simonov, Michael, Li, Darrick K., Tsay, Cynthia, Kawamura, Yuki, Shen, Matthew, Hsiao, Allen, Sekhon, Jasjeet S., Laine, Loren
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.11.2024
• Subjects: Electronic Health Record; Gastrointestinal Hemorrhage; Machine Learning; GIB; Gastrointestinal Hemorrhage; AI-CDSS; GBS; Machine Learning; EHR; Electronic Health Record; AUC
• ISSN: 0016-5085; 1528-0012; 1528-0012
• DOI: 10.1053/j.gastro.2024.06.030

Guidelines recommend use of risk stratification scores for patients presenting with gastrointestinal bleeding (GIB) to identify very-low-risk patients eligible for discharge from emergency departments. Machine learning models may outperform existing scores and can be integrated within the electronic health record (EHR) to provide real-time risk assessment without manual data entry. We present the first EHR-based machine learning model for GIB. The training cohort comprised 2546 patients and internal validation of 850 patients presenting with overt GIB (ie, hematemesis, melena, and hematochezia) to emergency departments of 2 hospitals from 2014 to 2019. External validation was performed on 926 patients presenting to a different hospital with the same EHR from 2014 to 2019. The primary outcome was a composite of red blood cell transfusion, hemostatic intervention (ie, endoscopic, interventional radiologic, or surgical), and 30-day all-cause mortality. We used structured data fields in the EHR, available within 4 hours of presentation, and compared the performance of machine learning models with current guideline-recommended risk scores, Glasgow-Blatchford Score, and Oakland Score. Primary analysis was area under the receiver operating characteristic curve. Secondary analysis was specificity at 99% sensitivity to assess the proportion of patients correctly identified as very low risk. The machine learning model outperformed the Glasgow-Blatchford Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001) and Oakland Score (area under the receiver operating characteristic curve, 0.92 vs 0.89; P < .001). At the very-low-risk threshold of 99% sensitivity, the machine learning model identified more very-low-risk patients: 37.9% vs 18.5% for Glasgow-Blatchford Score and 11.7% for Oakland Score (P < .001 for both comparisons). An EHR-based machine learning model performs better than currently recommended clinical risk scores and identifies more very-low-risk patients eligible for discharge from the emergency department. [Display omitted] Electronic health record–based machine learning models can provide automated, interpretable risk stratification for acute gastrointestinal bleeding to identify very-low-risk patients who could benefit from outpatient management.