Evaluating and Enhancing Large Language Models’ Performance in Domain-Specific Medicine: Development and Usability Study With DocOA

• Published in: Journal of medical Internet research Vol. 26; no. 1; p. e58158
• Main Authors: Chen, Xi, Wang, Li, You, MingKe, Liu, WeiZhi, Fu, Yu, Xu, Jie, Zhang, Shaoting, Chen, Gang, Li, Kang, Li, Jian
• Format: Journal Article
• Language: English
• Published: Toronto, Canada Journal of Medical Internet Research 22.07.2024; JMIR Publications
• Subjects: Case studies; Evidence-based medicine; Original Paper; Osteoarthritis
• ISSN: 1438-8871; 1439-4456; 1438-8871
• DOI: 10.2196/58158

Background The efficacy of large language models (LLMs) in domain-specific medicine, particularly for managing complex diseases such as osteoarthritis (OA), remains largely unexplored. Objective This study focused on evaluating and enhancing the clinical capabilities and explainability of LLMs in specific domains, using OA management as a case study. Methods A domain-specific benchmark framework was developed to evaluate LLMs across a spectrum from domain-specific knowledge to clinical applications in real-world clinical scenarios. DocOA, a specialized LLM designed for OA management integrating retrieval-augmented generation and instructional prompts, was developed. It can identify the clinical evidence upon which its answers are based through retrieval-augmented generation, thereby demonstrating the explainability of those answers. The study compared the performance of GPT-3.5, GPT-4, and a specialized assistant, DocOA, using objective and human evaluations. Results Results showed that general LLMs such as GPT-3.5 and GPT-4 were less effective in the specialized domain of OA management, particularly in providing personalized treatment recommendations. However, DocOA showed significant improvements. Conclusions This study introduces a novel benchmark framework that assesses the domain-specific abilities of LLMs in multiple aspects, highlights the limitations of generalized LLMs in clinical contexts, and demonstrates the potential of tailored approaches for developing domain-specific medical LLMs.