Comparing Artificial Intelligence–Generated and Clinician-Created Personalized Self-Management Guidance for Patients With Knee Osteoarthritis: Blinded Observational Study

• Published in: Journal of medical Internet research Vol. 27; no. 6; p. e67830
• Main Authors: Du, Kai, Li, Ao, Zuo, Qi-Heng, Zhang, Chen-Yu, Guo, Ren, Chen, Ping, Du, Wei-Shuai, Li, Shu-Ming
• Format: Journal Article
• Language: English
• Published: Canada Journal of Medical Internet Research 07.05.2025; Gunther Eysenbach MD MPH, Associate Professor; JMIR Publications
• Subjects: Academic guidance counseling; Accuracy; Aged; Analysis; Artificial; Artificial intelligence; Candidates; Care and treatment; Chatbots; Chronic illnesses; Clinical decision making; Clinical medicine; Clinical research; Content creation; Customization; Delineation; Disease; Disease management; Double-Blind Method; Efficiency; Evidence-based medicine; Female; Generative Artificial Intelligence; Health education; Health information; Health literacy; Humans; Knee; Language; Large language models; Male; Medical history; Middle Aged; Misinformation; Mobility; Natural language; Original Paper; Orthopedics; Osteoarthritis; Osteoarthritis, Knee - therapy; Oversight; Patient compliance; Patient education; Patient Education as Topic - methods; Performance evaluation; Precision Medicine; Privacy; Quality of life; Readability; Safeguards; Safety; Self-Management; Selfmanagement; Transparency; LLMs; osteoarthritis; orthopedics; self-management; artificial intelligence; observational study; personalized medicine; knee; ChatGPT; knee osteoarthritis; patient education; large language models; GPT-4; artificial intelligence in health care; AI-generated
• DOI: 10.2196/67830
• ISSN: 1438-8871; 1439-4456

Knee osteoarthritis is a prevalent, chronic musculoskeletal disorder that impairs mobility and quality of life. Personalized patient education aims to improve self-management and adherence; yet, its delivery is often limited by time constraints, clinician workload, and the heterogeneity of patient needs. Recent advances in large language models offer potential solutions. GPT-4 (OpenAI), distinguished by its long-context reasoning and adoption in clinical artificial intelligence research, emerged as a leading candidate for personalized health communication. However, its application in generating condition-specific educational guidance remains underexplored, and concerns about misinformation, personalization limits, and ethical oversight remain. We evaluated GPT-4's ability to generate individualized self-management guidance for patients with knee osteoarthritis in comparison with clinician-created content. This 2-phase, double-blind, observational study used data from 50 patients previously enrolled in a registered randomized trial. In phase 1, 2 orthopedic clinicians each generated personalized education materials for 25 patient profiles using anonymized clinical data, including history, symptoms, and lifestyle. In phase 2, the same datasets were processed by GPT-4 using standardized prompts. All content was anonymized and evaluated by 2 independent, blinded clinical experts using validated scoring systems. Evaluation criteria included efficiency, readability (Flesch-Kincaid, Gunning Fog, Coleman-Liau, and Simple Measure of Gobbledygook), accuracy, personalization, and comprehensiveness and safety. Disagreements between reviewers were resolved through consensus or third-party adjudication. GPT-4 outperformed clinicians in content generation speed (530.03 vs 37.29 words per min, P<.001). Readability was better on the Flesch-Kincaid (mean 11.56, SD 1.08 vs mean 12.67 SD 0.95), Gunning Fog (mean 12.47, SD 1.36 vs mean 14.56, SD 0.93), and Simple Measure of Gobbledygook (mean 13.33, SD 1.00 vs mean 13.81 SD 0.69) indices (all P<.001), though GPT-4 scored slightly higher on the Coleman-Liau Index (mean 15.90, SD 1.03 vs mean 15.15, SD 0.91). GPT-4 also outperformed clinicians in accuracy (mean 5.31, SD 1.73 vs mean 4.76, SD 1.10; P=.05, personalization (mean 54.32, SD 6.21 vs mean 33.20, SD 5.40; P<.001), comprehensiveness (mean 51.74, SD 6.47 vs mean 35.26, SD 6.66; P<.001), and safety (median 61, IQR 58-66 vs median 50, IQR 47-55.25; P<.001). GPT-4 could generate personalized self-management guidance for knee osteoarthritis with greater efficiency, accuracy, personalization, comprehensiveness, and safety than clinician-generated content, as assessed using standardized, guideline-aligned evaluation frameworks. These findings underscore the potential of large language models to support scalable, high-quality patient education in chronic disease management. The observed lexical complexity suggests the need to refine outputs for populations with limited health literacy. As an exploratory, single-center study, these results warrant confirmation in larger, multicenter cohorts with diverse demographic profiles. Future implementation should be guided by ethical and operational safeguards, including data privacy, transparency, and the delineation of clinical responsibility. Hybrid models integrating artificial intelligence-generated content with clinician oversight may offer a pragmatic path forward.