The limits of fair medical imaging AI in real-world generalization

• Published in: Nature medicine Vol. 30; no. 10; pp. 2838 - 2848
• Main Authors: Yang, Yuzhe, Zhang, Haoran, Gichoya, Judy W., Katabi, Dina, Ghassemi, Marzyeh
• Format: Journal Article
• Language: English
• Published: New York Nature Publishing Group US 01.10.2024; Nature Publishing Group
• Subjects: 692/700/139; 692/700/1421/1770; Algorithms; Artificial Intelligence; Best practice; Biomedical and Life Sciences; Biomedicine; Cancer Research; Chest; Coding; Demographics; Demography; Dermatology; Diagnostic Imaging - methods; Eye diseases; Female; Human performance; Humans; Infectious Diseases; Male; Medical electronics; Medical imaging; Metabolic Diseases; Molecular Medicine; Neurosciences; Ophthalmology; Optimization; Performance evaluation; Populations; Radiography, Thoracic; Radiology; Subpopulations; Training; X ray imagery
• ISSN: 1078-8956; 1546-170X; 1546-170X
• DOI: 10.1038/s41591-024-03113-4

As artificial intelligence (AI) rapidly approaches human-level performance in medical imaging, it is crucial that it does not exacerbate or propagate healthcare disparities. Previous research established AI’s capacity to infer demographic data from chest X-rays, leading to a key concern: do models using demographic shortcuts have unfair predictions across subpopulations? In this study, we conducted a thorough investigation into the extent to which medical AI uses demographic encodings, focusing on potential fairness discrepancies within both in-distribution training sets and external test sets. Our analysis covers three key medical imaging disciplines—radiology, dermatology and ophthalmology—and incorporates data from six global chest X-ray datasets. We confirm that medical imaging AI leverages demographic shortcuts in disease classification. Although correcting shortcuts algorithmically effectively addresses fairness gaps to create ‘locally optimal’ models within the original data distribution, this optimality is not true in new test settings. Surprisingly, we found that models with less encoding of demographic attributes are often most ‘globally optimal’, exhibiting better fairness during model evaluation in new test environments. Our work establishes best practices for medical imaging models that maintain their performance and fairness in deployments beyond their initial training contexts, underscoring critical considerations for AI clinical deployments across populations and sites. When tested across tasks, diseases and imaging modalities, the performance of AI models depends on encoding of demographic shortcuts, and correcting for them decreases their ability to generalize in new populations.