Clinical Information Extraction with Large Language Models: A Case Study on Organ Procurement

• Published in: AMIA ... Annual Symposium proceedings Vol. 2024; p. 115
• Main Authors: Adam, Hammaad, Lin, Junjing, Lin, Jianchang, Keenan, Hillary, Wilson, Ashia, Ghassemi, Marzyeh
• Format: Journal Article
• Language: English
• Published: United States 2024
• Subjects: Humans; Information Storage and Retrieval - methods; Large Language Models; Natural Language Processing; Tissue and Organ Procurement
• ISSN: 1942-597X; 1559-4076

Recent work has demonstrated that large language models (LLMs) are powerful tools for clinical information extraction from unstructured text. However, existing approaches have largely ignored the extraction of numeric information such as laboratory tests and vital signs. In this article, we present a case study on organ procurement that evaluates the ability of LLMs to extract numeric data from clinical text. We first describe our LLM-based approach, introducing a prompting strategy for numeric extraction and novel heuristics to combat hallucination. We validate our approach on a hand-annotated set of 298 notes, demonstrating that it has high accuracy, precision and recall. We then highlight the value of our approach for downstream data analysis using a corpus of 43,719 notes on 14,342 potential organ donors. This case study is a key component of an ongoing collaboration that aims to make data on organ procurement publicly available for informatics research.