LLM-Generated multiple choice practice quizzes for preclinical medical students

• Published in: Advances in physiology education Vol. 49; no. 3; pp. 758 - 763
• Main Authors: Camarata, Troy, McCoy, Lise, Rosenberg, Robert, Temprine Grellinger, Kelsey R., Brettschnieder, Kylie, Berman, Jonathan
• Format: Journal Article
• Language: English
• Published: United States 01.09.2025
• Subjects: Education, Medical, Undergraduate - methods; Education, Medical, Undergraduate - standards; Educational Measurement - methods; Educational Measurement - standards; Feasibility Studies; Humans; Students, Medical; multiple choice questions; medical education; large language model; practice questions; artificial intelligence
• ISSN: 1043-4046; 1522-1229
• DOI: 10.1152/advan.00106.2024

Practice board exam questions generated by large language models can be made suitable for preclinical medical students by subject-matter experts. Multiple choice questions (MCQs) are frequently used in medical education for assessment. Automated generation of MCQs in board-exam format could potentially save significant effort for faculty and generate a wider set of practice materials for student use. The goal of this study was to explore the feasibility of using ChatGPT by OpenAI to generate United States Medical Licensing Exam (USMLE)/Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA)-style practice quiz items as study aids. Researchers gave second-year medical students studying renal physiology access to a set of practice quizzes with ChatGPT-generated questions. The exam items generated were evaluated by independent experts for quality and adherence to the National Board of Medical Examiners (NBME)/National Board of Osteopathic Medical Examiners (NBOME) guidelines. Forty-nine percent of questions contained item writing flaws, and 22% contained factual or conceptual errors. However, 59/65 (91%) were categorized as a reasonable starting point for revision. These results demonstrate the feasibility of large language model (LLM)-generated practice questions in medical education but only when supervised by a subject matter expert with training in exam item writing. NEW & NOTEWORTHY Practice board exam questions generated by large language models can be made suitable for preclinical medical students by subject-matter experts.