Even with ChatGPT, race matters

• Published in: Clinical imaging Vol. 109; p. 110113
• Main Authors: Amin, Kanhai S., Forman, Howard P., Davis, Melissa A.
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.05.2024
• Subjects: ChatGPT; Health equity; Humans; Implicit bias; Language; Large language models; Radiology; Radiology report; United States; United States; Health equity; Large language models; ChatGPT; Radiology report; Implicit bias
• ISSN: 0899-7071; 1873-4499; 1873-4499
• DOI: 10.1016/j.clinimag.2024.110113

Applications of large language models such as ChatGPT are increasingly being studied. Before these technologies become entrenched, it is crucial to analyze whether they perpetuate racial inequities. We asked Open AI's ChatGPT-3.5 and ChatGPT-4 to simplify 750 radiology reports with the prompt “I am a ___ patient. Simplify this radiology report:” while providing the context of the five major racial classifications on the U.S. census: White, Black or African American, American Indian or Alaska Native, Asian, and Native Hawaiian or other Pacific Islander. To ensure an unbiased analysis, the readability scores of the outputs were calculated and compared. Statistically significant differences were found in both models based on the racial context. For ChatGPT-3.5, output for White and Asian was at a significantly higher reading grade level than both Black or African American and American Indian or Alaska Native, among other differences. For ChatGPT-4, output for Asian was at a significantly higher reading grade level than American Indian or Alaska Native and Native Hawaiian or other Pacific Islander, among other differences. Here, we tested an application where we would expect no differences in output based on racial classification. Hence, the differences found are alarming and demonstrate that the medical community must remain vigilant to ensure large language models do not provide biased or otherwise harmful outputs. •ChatGPT can significantly simplify radiology reports.•Level of simplification significantly differed based on racial context.•Reports for White and Asian patients were simplified at a higher reading grade level than for Black/African American and American Indian/Alaska Native patients.