ChatGPT efficacy for answering musculoskeletal anatomy questions: a study evaluating quality and consistency between raters and timepoints

• Published in: Surgical and radiologic anatomy (English ed.) Vol. 46; no. 11; pp. 1885 - 1890
• Main Authors: Mantzou, Nikolaos, Ediaroglou, Vasileios, Drakonaki, Elena, Syggelos, Spyros A., Karageorgos, Filippos F., Totlis, Trifon
• Format: Journal Article
• Language: English
• Published: Paris Springer Paris 01.11.2024; Springer Nature B.V
• Subjects: Anatomy; Anatomy - education; Chatbots; Computer-Assisted Instruction - methods; Educational Measurement; Humans; Imaging; Likert scale; Medicine; Medicine & Public Health; Musculoskeletal System - anatomy & histology; Original Article; Orthopedics; Radiology; Reproducibility of Results; Surgery; Large language models; Anatomy; ChatGPT; Artificial intelligence
• ISSN: 1279-8517; 0930-1038; 1279-8517
• DOI: 10.1007/s00276-024-03477-9

Purpose There is increasing interest in the use of digital platforms such as ChatGPT for anatomy education. This study aims to evaluate the efficacy of ChatGPT in providing accurate and consistent responses to questions focusing on musculoskeletal anatomy across various time points (hours and days). Methods A selection of 6 Anatomy-related questions were asked to ChatGPT 3.5 in 4 different timepoints. All answers were rated blindly by 3 expert raters for quality according to a 5 -point Likert Scale. Difference of 0 or 1 points in Likert scale scores between raters was considered as agreement and between different timepoints was considered as consistent indicating good reproducibility. Results There was significant variation in the quality of the answers ranging from extremely good to very poor quality. There was also variation of consistency levels between different timepoints. Answers were rated as good quality ( ≥  3 in Likert scale) in 50% of cases (3/6) and as consistent in 66.6% (4/6) of cases. In the low-quality answers, significant mistakes, conflicting data or lack of information were encountered. Conclusion As of the time of this article, the quality and consistency of the ChatGPT v3.5 answers is variable, thus limiting its utility as independent and reliable resource of learning musculoskeletal anatomy. Validating information by reviewing the anatomical literature is highly recommended.