Comparison of ChatGPT versions in informing patients with rotator cuff injuries

• Published in: JSES international Vol. 8; no. 5; pp. 1016 - 1018
• Main Authors: Günay, Ali Eray, Özer, Alper, Yazıcı, Alparslan, Sayer, Gökhan
• Format: Journal Article
• Language: English
• Published: United States Elsevier Inc 01.09.2024; Elsevier
• Subjects: Arthroscopy; Artificial intelligence; ChatGPT; OpenAI; Rotator cuff; Shoulder; Arthroscopy; ChatGPT; Shoulder; Basic Science Study; OpenAI; Validation of AI Programs; Artificial intelligence; Rotator cuff
• ISSN: 2666-6383; 2666-6383
• DOI: 10.1016/j.jseint.2024.04.016

The aim of this study is to evaluate whether Chat Generative Pretrained Transformer (ChatGPT) can be recommended as a resource for informing patients planning rotator cuff repairs, and to assess the differences between ChatGPT 3.5 and 4.0 versions in terms of information content and readability. In August 2023, 13 commonly asked questions by patients with rotator cuff disease were posed to ChatGPT 3.5 and ChatGPT 4 programs using different internet protocol computers by 3 experienced surgeons in rotator cuff surgery. After converting the answers of both versions into text, the quality and readability of the answers were examined. The average Journal of the American Medical Association score for both versions was 0, and the average DISCERN score was 61.6. A statistically significant and strong correlation was found between ChatGPT 3.5 and 4.0 DISCERN scores. There was excellent agreement in DISCERN scores for both versions among the 3 evaluators. ChatGPT 3.5 was found to be less readable than ChatGPT 4.0. The information provided by the ChatGPT conversational system was evaluated as of high quality, but there were significant shortcomings in terms of reliability due to the lack of citations. Despite the ChatGPT 4.0 version having higher readability scores, both versions were considered difficult to read.