The impact of class imbalance in classification performance metrics based on the binary confusion matrix

• Published in: Pattern recognition Vol. 91; pp. 216 - 231
• Main Authors: Luque, Amalia, Carrasco, Alejandro, Martín, Alejandro, de las Heras, Ana
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.07.2019
• Subjects: Class Balance Metrics; Classification; Imbalanced datasets; Performance measures; Class Balance Metrics; Performance measures; Imbalanced datasets; Classification
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2019.02.023

•Imbalance coefficient fosters measuring imbalance.•Geometric Mean and Bookmaker Informedness constitute the best unbiased metrics.•Matthews Correlation Coefficient is the best option for error consideration.•The concept of Class Balance Accuracy can be extended to other metrics. A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other well-known performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of null-biased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics.