Comparative Analysis of Accuracy, Readability, Sentiment, and Actionability: Artificial Intelligence Chatbots (ChatGPT and Google Gemini) versus Traditional Patient Information Leaflets for Local Anesthesia in Eye Surgery

• Published in: British and Irish orthoptic journal Vol. 20; no. 1; pp. 183 - 192
• Main Authors: Gondode, Prakash, Duggal, Sakshi, Garg, Neha, Lohakare, Pooja, Jakhar, Jubin, Bharti, Swati, Dewangan, Shraddha
• Format: Journal Article
• Language: English
• Published: England Ubiquity Press Ltd 2024; Ubiquity Press; White Rose University Press
• Subjects: Accuracy; Age groups; ai artificial intelligence; Analysis; Anesthesia; Artificial intelligence; cataract; Caterers and catering; Chatbots; Comparative analysis; Computational linguistics; Confidentiality; Empowerment; Eye; Eye surgery; Human subjects; Information services; Language; Language processing; Large language models; Local anesthesia; local anesthetic; Medical advice systems; Natural language; Natural language interfaces; Online information services; Online services; Patient education; patient education handout; Professional ethics; Professionals; Readability; Sentiment analysis; Surgery; Surgical outcomes; Variance analysis; Local anesthetic; Cataract; AI Artificial Intelligence; Readability; Patient education handout
• ISSN: 2516-3590; 1743-9868; 2516-3590
• DOI: 10.22599/bioj.377

Eye surgeries often evoke strong negative emotions in patients, including fear and anxiety. Patient education material plays a crucial role in informing and empowering individuals. Traditional sources of medical information may not effectively address individual patient concerns or cater to varying levels of understanding. This study aims to conduct a comparative analysis of the accuracy, completeness, readability, tone, and understandability of patient education material generated by AI chatbots versus traditional Patient Information Leaflets (PILs), focusing on local anesthesia in eye surgery. Expert reviewers evaluated responses generated by AI chatbots (ChatGPT and Google Gemini) and a traditional PIL (Royal College of Anaesthetists' PIL) based on accuracy, completeness, readability, sentiment, and understandability. Statistical analyses, including ANOVA and Tukey HSD tests, were conducted to compare the performance of the sources. Readability analysis showed variations in complexity among the sources, with AI chatbots offering simplified language and PILs maintaining better overall readability and accessibility. Sentiment analysis revealed differences in emotional tone, with Google Gemini exhibiting the most positive sentiment. AI chatbots demonstrated superior understandability and actionability, while PILs excelled in completeness. Overall, ChatGPT showed slightly higher accuracy (scores expressed as mean ± standard deviation) (4.71 ± 0.5 vs 4.61 ± 0.62) and completeness (4.55 ± 0.58 vs 4.47 ± 0.58) compared to Google Gemini, but PILs performed best (4.84 ± 0.37 vs 4.88 ± 0.33) in terms of both accuracy and completeness (p-value for completeness <0.05). AI chatbots show promise as innovative tools for patient education, complementing traditional PILs. By leveraging the strengths of both AI-driven technologies and human expertise, healthcare providers can enhance patient education and empower individuals to make informed decisions about their health and medical care.