Assessing the discriminative ability of risk models for more than two outcome categories

• Published in: European journal of epidemiology Vol. 27; no. 10; pp. 761 - 770
• Main Authors: Van Calster, Ben, Vergouwe, Yvonne, Looman, Caspar W. N., Van Belle, Vanya, Timmerman, Dirk, Steyerberg, Ewout W.
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer 01.10.2012; Springer Netherlands; Springer Nature B.V
• Subjects: Benign neoplasms; Biological and medical sciences; Cancer; Cardiology; Data Interpretation, Statistical; Discriminant Analysis; Epidemiology; Female; General aspects; Health outcomes; Humans; Infectious Diseases; Logistic Models; Logistic regression; Male; Medical sciences; Medicine; Medicine & Public Health; METHODS; Miscellaneous; Modeling; Models, Statistical; Oncology; Ovarian cancer; Ovarian Neoplasms - diagnosis; Ovarian Neoplasms - epidemiology; Prediction models; Predictions; Prevalence; Prognosis; Public Health; Public health. Hygiene; Public health. Hygiene-occupational medicine; Regression analysis; Residual neoplasm; Risk; Risk Assessment; ROC Curve; Teratoma; Testicular Neoplasms - diagnosis; Testicular Neoplasms - epidemiology; Tumors; index; Polytomous risk prediction; Discrimination; Discrimination plot; Ability; Human; Evaluation; Prognosis; c-index; Prediction; Risk; Index; Epidemiology; Risk factor; Evolution; Models; Predictive factor; Public health
• ISSN: 0393-2990; 1573-7284
• DOI: 10.1007/s10654-012-9733-3

The discriminative ability of risk models for dichotomous outcomes is often evaluated with the concordance index (c-index). However, many medical prediction problems are polytomous, meaning that more than two outcome categories need to be predicted. Unfortunately such problems are often dichotomized in prediction research. We present a perspective on the evaluation of discriminative ability of polytomous risk models, which may instigate researchers to consider polytomous prediction models more often. First, we suggest a "discrimination plot" as a tool to visualize the model's discriminative ability. Second, we discuss the use of one overall polytomous c-index versus a set of dichotomous measures to summarize the performance of the model. Third, we address several aspects to consider when constructing a polytomous c-index. These involve the assessment of concordance in pairs versus sets of patients, weighting by outcome prevalence, the value related to models with random performance, the reduction to the dichotomous c-index for dichotomous problems, and interpretation. We illustrate these issues on case studies dealing with ovarian cancer (four outcome categories) and testicular cancer (three categories). We recommend the use of a discrimination plot together with an overall c-index such as the Polytomous Discrimination Index. If the overall c-index suggests that the model has relevant discriminative ability, pairwise c-indexes for each pair of outcome categories are informative. For pairwise c-indexes we recommend the 'conditional-risk' method which is consistent with the analytical approach of the multinomial logistic regression used to develop polytomous risk models.