Classifying Unstructured Text in Electronic Health Records for Mental Health Prediction Models: Large Language Model Evaluation Study

• Published in: JMIR medical informatics Vol. 13; p. e65454
• Main Authors: Cardamone, Nicholas C, Olfson, Mark, Schmutte, Timothy, Ungar, Lyle, Liu, Tony, Cullen, Sara W, Williams, Nathaniel J, Marcus, Steven C
• Format: Journal Article
• Language: English
• Published: Canada JMIR Publications 21.01.2025
• Subjects: AI Language Models in Health Care; Anxiety; Artificial intelligence; Chronic fatigue syndrome; Classification; Data warehouses; Datasets; Decision Support for Health Professionals; Diagnostic Tools in Mental Health; Disease; Eating disorders; Electronic Health Records; Electronic Health Records - statistics & numerical data; Emergency Service, Hospital; Emotional disorders; Humans; Impulsivity; Large Language Models; Machine Learning; Medical coding; Mental disorders; Mental Disorders - diagnosis; Mental Health; Methods and New Tools in Mental Health Research; Mood disorders; Multimedia; Natural Language Processing; New Technologies; Original Paper; Personality disorders; Psychosis; Reconciliation; Review boards; Self destructive behavior; United States; United States; natural language processing; mental health; AI; mental health disorder; LLM; emergency department; large language model; machine learning; EHR; EHR system; artificial intelligence; NLP; physical health; ChatGPT; predictive modeling; health informatics; text; dataset; electronic health record; ML
• DOI: 10.2196/65454
• ISSN: 2291-9694

Prediction models have demonstrated a range of applications across medicine, including using electronic health record (EHR) data to identify hospital readmission and mortality risk. Large language models (LLMs) can transform unstructured EHR text into structured features, which can then be integrated into statistical prediction models, ensuring that the results are both clinically meaningful and interpretable. This study aims to compare the classification decisions made by clinical experts with those generated by a state-of-the-art LLM, using terms extracted from a large EHR data set of individuals with mental health disorders seen in emergency departments (EDs). Using a dataset from the EHR systems of more than 50 health care provider organizations in the United States from 2016 to 2021, we extracted all clinical terms that appeared in at least 1000 records of individuals admitted to the ED for a mental health-related problem from a source population of over 6 million ED episodes. Two experienced mental health clinicians (one medically trained psychiatrist and one clinical psychologist) reached consensus on the classification of EHR terms and diagnostic codes into categories. We evaluated an LLM's agreement with clinical judgment across three classification tasks as follows: (1) classify terms into "mental health" or "physical health", (2) classify mental health terms into 1 of 42 prespecified categories, and (3) classify physical health terms into 1 of 19 prespecified broad categories. There was high agreement between the LLM and clinical experts when categorizing 4553 terms as "mental health" or "physical health" (κ=0.77, 95% CI 0.75-0.80). However, there was still considerable variability in LLM-clinician agreement on the classification of mental health terms (κ=0.62, 95% CI 0.59-0.66) and physical health terms (κ=0.69, 95% CI 0.67-0.70). The LLM displayed high agreement with clinical experts when classifying EHR terms into certain mental health or physical health term categories. However, agreement with clinical experts varied considerably within both sets of mental and physical health term categories. Importantly, the use of LLMs presents an alternative to manual human coding, presenting great potential to create interpretable features for prediction models.