Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics

Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI m...

Full description

Saved in:
Bibliographic Details
Published inJournal of medical Internet research Vol. 27; no. 7956; p. e70901
Main Authors Sun, Luning, Gibbons, Christopher, Hernández-Orallo, José, Wang, Xiting, Jiang, Liming, Stillwell, David, Luo, Fang, Xie, Xing
Format Journal Article
LanguageEnglish
Published Canada Journal of Medical Internet Research 26.05.2025
Gunther Eysenbach MD MPH, Associate Professor
JMIR Publications
Subjects
Accuracy
Applications of AI
Artificial Intelligence
Benchmarking
Benchmarks
Chatbots
Cognition & reasoning
Cognitive ability
Development and Evaluation of Research Methods, Instruments and Tools
Empathy
Health care
Humans
Large language models
Licenses
Licensing examinations
Medical practices
Methods
Personality
Physicians
Power
Program Evaluation and Review Technique
Psychometrics
Quantitative psychology
Subject specialists
Verbal communication
Viewpoint
Viewpoints and s
Work skills
United States
United Kingdom > UK
generalist medical artificial intelligence
human oversight
predictive power
construct-oriented evaluation
data contamination
explanatory power
psychometrics
benchmark
health care
Online AccessGet full text

Cover

More Information
Summary:Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
CG is an employee of Oracle Health Inc., serves on the Board of Directors at the International Society for Quality of Life Research, and holds stock in Oracle Corporation. XW has previously been employed at Microsoft Research and holds stock in Microsoft. LJ has previously served as an intern at Microsoft Research. XX is an employee of Microsoft Research and holds stock in Microsoft. All other authors declare no conflicts of interest.
these authors contributed equally
ISSN:1438-8871
1439-4456
1438-8871
DOI:10.2196/70901