Proof of biased behavior of Normalized Mutual Information

• Published in: Scientific reports Vol. 14; no. 1; pp. 9021 - 17
• Main Authors: Mahmoudi, Amin, Jemielniak, Dariusz
• Format: Journal Article
• Language: English
• Published: London Nature Publishing Group UK 19.04.2024; Nature Publishing Group; Nature Portfolio
• Subjects: 639/705/1042; 639/705/117; Accuracy; Algorithms; Bias; Case studies; Clustering; False information; Humanities and Social Sciences; Lattice theory; Modularity; multidisciplinary; Science; Science (multidisciplinary); Social networks
• ISSN: 2045-2322; 2045-2322
• DOI: 10.1038/s41598-024-59073-9

The Normalized Mutual Information (NMI) metric is widely utilized in the evaluation of clustering and community detection algorithms. This study explores the performance of NMI, specifically examining its performance in relation to the quantity of communities, and uncovers a significant drawback associated with the metric's behavior as the number of communities increases. Our findings reveal a pronounced bias in the NMI as the number of communities escalates. While previous studies have noted this biased behavior, they have not provided a formal proof and have not addressed the causation of this problem, leaving a gap in the existing literature. In this study, we fill this gap by employing a mathematical approach to formally demonstrate why NMI exhibits biased behavior, thereby establishing its unsuitability as a metric for evaluating clustering and community detection algorithms. Crucially, our study exposes the vulnerability of entropy-based metrics that employ logarithmic functions to similar bias.