powerROC: An Interactive Web Tool for Sample Size Calculation in Assessing Models' Discriminative Abilities

• Published in: AMIA Summits on Translational Science proceedings Vol. 2025; pp. 196 - 204
• Main Authors: Grolleau, François, Tibshirani, Robert, Chen, Jonathan H
• Format: Journal Article
• Language: English
• Published: United States American Medical Informatics Association 2025
• ISSN: 2153-4063; 2153-4063

Rigorous external validation is crucial for assessing the generalizability of prediction models, particularly by evaluating their discrimination (AUROC) on new data. This often involves comparing a new model's AUROC to that of an established reference model. However, many studies rely on arbitrary rules of thumb for sample size calculations, often resulting in underpowered analyses and unreliable conclusions. This paper reviews crucial concepts for accurate sample size determination in AUROC-based external validation studies, making the theory and practice more accessible to researchers and clinicians. We introduce powerROC, an open-source web tool designed to simplify these calculations, enabling both the evaluation of a single model and the comparison of two models. The tool offers guidance on selecting target precision levels and employs flexible approaches, leveraging either pilot data or user-defined probability distributions. We illustrate powerROC's utility through a case study on hospital mortality prediction using the MIMIC database.