How well do large language model-based chatbots perform in oral and maxillofacial radiology?

This study evaluated the performance of four large language model (LLM)-based chatbots by comparing their test results with those of dental students on an oral and maxillofacial radiology examination. ChatGPT, ChatGPT Plus, Bard, and Bing Chat were tested on 52 questions from regular dental college...

Full description

Saved in:
Bibliographic Details
Published inDento-maxillo-facial radiology Vol. 53; no. 6; p. 390
Main Authors Jeong, Hui, Han, Sang-Sun, Yu, Youngjae, Kim, Saejin, Jeon, Kug Jin
Format Journal Article
LanguageEnglish
Published England 01.09.2024
Subjects
Education, Dental - methods
Educational Measurement - methods
Humans
Language
Radiography, Dental - methods
Radiology - education
chatbot
education
large language model
dental
artificial intelligence
oral and maxillofacial radiology
Online AccessGet more information

Cover

More Information
Summary:This study evaluated the performance of four large language model (LLM)-based chatbots by comparing their test results with those of dental students on an oral and maxillofacial radiology examination. ChatGPT, ChatGPT Plus, Bard, and Bing Chat were tested on 52 questions from regular dental college examinations. These questions were categorized into three educational content areas: basic knowledge, imaging and equipment, and image interpretation. They were also classified as multiple-choice questions (MCQs) and short-answer questions (SAQs). The accuracy rates of the chatbots were compared with the performance of students, and further analysis was conducted based on the educational content and question type. The students' overall accuracy rate was 81.2%, while that of the chatbots varied: 50.0% for ChatGPT, 65.4% for ChatGPT Plus, 50.0% for Bard, and 63.5% for Bing Chat. ChatGPT Plus achieved a higher accuracy rate for basic knowledge than the students (93.8% vs. 78.7%). However, all chatbots performed poorly in image interpretation, with accuracy rates below 35.0%. All chatbots scored less than 60.0% on MCQs, but performed better on SAQs. The performance of chatbots in oral and maxillofacial radiology was unsatisfactory. Further training using specific, relevant data derived solely from reliable sources is required. Additionally, the validity of these chatbots' responses must be meticulously verified.
ISSN:1476-542X
DOI:10.1093/dmfr/twae021