ChatGPT (GPT-4) versus doctors on complex cases of the Swedish family medicine specialist examination: an observational comparative study

• Published in: BMJ open Vol. 14; no. 12; p. e086148
• Main Authors: Arvidsson, Rasmus, Gunnarsson, Ronny, Entezarjou, Artin, Sundemo, David, Wikberg, Carl
• Format: Journal Article
• Language: English
• Published: England British Medical Journal Publishing Group 26.12.2024; BMJ Publishing Group LTD; BMJ Publishing Group
• Subjects: Allmänmedicin; Artificial Intelligence; Chatbots; Clinical Competence; Computer Systems; Computer Vision and learning System; Datorseende och lärande system; Datorsystem; Educational Measurement - methods; Family Practice; General Medicine; General practice / Family practice; Health informatics; Humans; Male; Medicine; Multiple choice; Original Research; Patients; Physicians; Primary care; Primary Health Care; Statistical analysis; Sweden; Sweden; Artificial Intelligence; Primary Health Care; Health informatics
• ISSN: 2044-6055; 2044-6055
• DOI: 10.1136/bmjopen-2024-086148

BackgroundRecent breakthroughs in artificial intelligence research include the development of generative pretrained transformers (GPT). ChatGPT has been shown to perform well when answering several sets of medical multiple-choice questions. However, it has not been tested for writing free-text assessments of complex cases in primary care.ObjectivesTo compare the performance of ChatGPT, version GPT-4, with that of real doctors.Design and settingA blinded observational comparative study conducted in the Swedish primary care setting. Responses from GPT-4 and real doctors to cases from the Swedish family medicine specialist examination were scored by blinded reviewers, and the scores were compared.ParticipantsAnonymous responses from the Swedish family medicine specialist examination 2017–2022 were used.Outcome measuresPrimary: the mean difference in scores between GPT-4’s responses and randomly selected responses by human doctors, as well as between GPT-4’s responses and top-tier responses by human doctors. Secondary: the correlation between differences in response length and response score; the intraclass correlation coefficient between reviewers; and the percentage of maximum score achieved by each group in different subject categories.ResultsThe mean scores were 6.0, 7.2 and 4.5 for randomly selected doctor responses, top-tier doctor responses and GPT-4 responses, respectively, on a 10-point scale. The scores for the random doctor responses were, on average, 1.6 points higher than those of GPT-4 (p<0.001, 95% CI 0.9 to 2.2) and the top-tier doctor scores were, on average, 2.7 points higher than those of GPT-4 (p<0.001, 95 % CI 2.2 to 3.3). Following the release of GPT-4o, the experiment was repeated, although this time with only a single reviewer scoring the answers. In this follow-up, random doctor responses were scored 0.7 points higher than those of GPT-4o (p=0.044).ConclusionIn complex primary care cases, GPT-4 performs worse than human doctors taking the family medicine specialist examination. Future GPT-based chatbots may perform better, but comprehensive evaluations are needed before implementing chatbots for medical decision support in primary care.