Performance Analysis of Three-Class Classifiers: Properties of a 3-D ROC Surface and the Normalized Volume Under the Surface for the Ideal Observer

• Published in: IEEE transactions on medical imaging Vol. 27; no. 2; pp. 215 - 227
• Main Authors: Sahiner, B., Heang-Ping Chan, Hadjiiski, L.M.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2008; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Area measurement; Artificial Intelligence; Biomedical imaging; Cancer; Covariance matrix; Ideal observer; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Imaging, Three-Dimensional - methods; Magnetic resonance imaging; Medical diagnostic imaging; Monte Carlo simulation; Observer Variation; Pattern recognition; Pattern Recognition, Automated - methods; Performance analysis; performance index; Radiology; receiver operating characteristic (ROC) analysis; Reproducibility of Results; ROC Curve; Sensitivity and Specificity; Studies; three-class classification; Volume measurement; 3-class classification; performance index; ideal observer; ROC analysis
• ISSN: 0278-0062; 1558-254X
• DOI: 10.1109/TMI.2007.905822

Classification of a given observation to one of three classes is an important task in many decision processes or pattern recognition applications. A general analysis of the performance of three-class classifiers results in a complex 6-D receiver operating characteristic (ROC) space, for which no simple analytical tool exists at present. We investigate the performance of an ideal observer under a specific set of assumptions that reduces the 6-D ROC space to 3-D by constraining the utilities of some of the decisions in the classification task. These assumptions lead to a 3-D ROC space in which the true-positive fraction (TPF) can be expressed in terms of the two types of false-positive fractions (FPFs). We demonstrate that the TPF is uniquely determined by, and therefore is a function of, the two FPFs. The domain of this function is shown to be related to the decision boundaries in the likelihood ratio plane. Based on these properties of the 3-D ROC space, we can define a summary measure, referred to as the normalized volume under the surface (NVUS), that is analogous to the area under the ROC curve (AUC) for a two-class classifier. We further investigate the properties of the 3-D ROC surface and the NVUS for the ideal observer under the condition that the three class distributions are multivariate normal with equal covariance matrices. The probability density functions (pdfs) of the decision variables are shown to follow a bivariate log-normal distribution. By considering these pdfs, we express the TPF in terms of the FPFs, and integrate the TPF over its domain numerically to obtain the NVUS. In addition, we performed a Monte Carlo simulation study, in which the 3-D ROC surface was generated by empirical "optimal" classification of case samples in the multidimensional feature space following the assumed distributions, to obtain an independent estimate of NVUS. The NVUS value obtained by using the analytical pdfs was found to be in good agreement with that obtained from the Monte Carlo simulation study. We also found that, under all conditions studied, the NVUS increased when the difficulty of the classification task was reduced by changing the parameters of the class distributions, thereby exhibiting the properties of a performance metric in analogous to AUC. Our results indicate that, under the conditions that lead to our 3-D ROC analysis, the performance of a three-class classifier may be analyzed by considering the ROC surface, and its accuracy characterized by the NVUS.