Estimating the Number of Clusters Using Cross-Validation

• Published in: Journal of computational and graphical statistics Vol. 29; no. 1; pp. 162 - 173
• Main Authors: Fu, Wei, Perry, Patrick O.
• Format: Journal Article
• Language: English
• Published: Alexandria Taylor & Francis 02.01.2020; Taylor & Francis Ltd
• Subjects: Cell cycle; Clustering; Computer simulation; Cross-validation; Data analysis; k-Means; Methods; Model selection; Unsupervised learning
• ISSN: 1061-8600; 1537-2715
• DOI: 10.1080/10618600.2019.1647846

Many clustering methods, including k-means, require the user to specify the number of clusters as an input parameter. A variety of methods have been devised to choose the number of clusters automatically, but they often rely on strong modeling assumptions. This article proposes a data-driven approach to estimate the number of clusters based on a novel form of cross-validation. The proposed method differs from ordinary cross-validation, because clustering is fundamentally an unsupervised learning problem. Simulation and real data analysis results show that the proposed method outperforms existing methods, especially in high-dimensional settings with heterogeneous or heavy-tailed noise. In a yeast cell cycle dataset, the proposed method finds a parsimonious clustering with interpretable gene groupings. Supplementary materials for this article are available online.