ChatGPT versus a customized AI chatbot (Anatbuddy) for anatomy education: A comparative pilot study

• Published in: Anatomical sciences education Vol. 17; no. 7; pp. 1396 - 1405
• Main Authors: Arun, Gautham, Perumal, Vivek, Urias, Francis Paul John Bato, Ler, Yan En, Tan, Bryan Wen Tao, Vallabhajosyula, Ranganath, Tan, Emmanuel, Ng, Olivia, Ng, Kian Bee, Mogali, Sreenivasulu Reddy
• Format: Journal Article
• Language: English
• Published: United States Wiley Subscription Services, Inc 01.10.2024
• Subjects: Accuracy; anatomical education; Anatomy; Anatomy - education; Artificial Intelligence; chatbot; Chatbots; ChatGPT; Computer-Assisted Instruction - methods; Educational Measurement - statistics & numerical data; Humans; Learning; personalized learning; Pilot Projects; retrieval augmented generation; Software; anatomical education; chatbot; retrieval augmented generation; ChatGPT; artificial intelligence; personalized learning
• ISSN: 1935-9772; 1935-9780; 1935-9780
• DOI: 10.1002/ase.2502

Large Language Models (LLMs) have the potential to improve education by personalizing learning. However, ChatGPT‐generated content has been criticized for sometimes producing false, biased, and/or hallucinatory information. To evaluate AI's ability to return clear and accurate anatomy information, this study generated a custom interactive and intelligent chatbot (Anatbuddy) through an Open AI Application Programming Interface (API) that enables seamless AI‐driven interactions within a secured cloud infrastructure. Anatbuddy was programmed through a Retrieval Augmented Generation (RAG) method to provide context‐aware responses to user queries based on a predetermined knowledge base. To compare their outputs, various queries (i.e., prompts) on thoracic anatomy (n = 18) were fed into Anatbuddy and ChatGPT 3.5. A panel comprising three experienced anatomists evaluated both tools' responses for factual accuracy, relevance, completeness, coherence, and fluency on a 5‐point Likert scale. These ratings were reviewed by a third party blinded to the study, who revised and finalized scores as needed. Anatbuddy's factual accuracy (mean ± SD = 4.78/5.00 ± 0.43; median = 5.00) was rated significantly higher (U = 84, p = 0.01) than ChatGPT's accuracy (4.11 ± 0.83; median = 4.00). No statistically significant differences were detected between the chatbots for the other variables. Given ChatGPT's current content knowledge limitations, we strongly recommend the anatomy profession develop a custom AI chatbot for anatomy education utilizing a carefully curated knowledge base to ensure accuracy. Further research is needed to determine students' acceptance of custom chatbots for anatomy education and their influence on learning experiences and outcomes.