To safely deploy generative AI in health care, models must be open source

• Published in: Nature (London) Vol. 624; no. 7990; pp. 36 - 38
• Main Authors: Toma, Augustin, Senkaiahliyan, Senthujan, Lawler, Patrick R, Rubin, Barry, Wang, Bo
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group 07.12.2023
• Subjects: Artificial intelligence; Chatbots; Collaboration; Conflicts of interest; Consortia; Datasets; Generative artificial intelligence; Health care; Health care facilities; Health services; Medical records; Medical research; Modelling; Patient safety; Privacy; Proprietary; Software; United Kingdom > UK; United States > US
• ISSN: 0028-0836; 1476-4687
• DOI: 10.1038/d41586-023-03803-y

Thanks to the two companies collaborating, initiatives are already under way at the University of California San Diego Health system and at Stanford University Medical Center in California. If people's medical records are used to train the model, it is possible that with the relevant prompts, the model could recreate and leak sensitive information13 - particularly if it is trained on data from people with a rare combination of medical conditionsorcharacteristics. Because the models are products of the vast swathes of data from the Internet that they are trained on, LLMs could exacerbate biases around gender, race, disability and socioeconomic status14. Another problem specific to proprietary LLMs is that companies' dependency on profits creates an inherent conflict of interest that could inject instability into the provision of medical care. All in all, it is hard to see how LLMs that are developed and controlled behind closed corporate doors could be broadly adopted in health care without underminingtheaccountability and transparency of both medical research and medical care.