Enhancing Small Medical Learners with Privacy-preserving Contextual Prompting

• Published in: arXiv.org
• Main Authors: Zhang, Xinlu, Li, Shiyang, Yang, Xianjun, Tian, Chenxin, Yao Qin, Petzold, Linda Ruth
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 16.05.2024
• Subjects: Context; Decision making; Harnesses; Large language models; Privacy
• ISSN: 2331-8422

Large language models (LLMs) demonstrate remarkable medical expertise, but data privacy concerns impede their direct use in healthcare environments. Although offering improved data privacy protection, domain-specific small language models (SLMs) often underperform LLMs, emphasizing the need for methods that reduce this performance gap while alleviating privacy concerns. In this paper, we present a simple yet effective method that harnesses LLMs' medical proficiency to boost SLM performance in medical tasks under privacy-restricted scenarios. Specifically, we mitigate patient privacy issues by extracting keywords from medical data and prompting the LLM to generate a medical knowledge-intensive context by simulating clinicians' thought processes. This context serves as additional input for SLMs, augmenting their decision-making capabilities. Our method significantly enhances performance in both few-shot and full training settings across three medical knowledge-intensive tasks, achieving up to a 22.57% increase in absolute accuracy compared to SLM fine-tuning without context, and sets new state-of-the-art results in two medical tasks within privacy-restricted scenarios. Further out-of-domain testing and experiments in two general domain datasets showcase its generalizability and broad applicability. Our code can be found at https://github.com/XZhang97666/PrivacyBoost-SLM.