Efficient set-valued prediction in multi-class classification

• Published in: Data mining and knowledge discovery Vol. 35; no. 4; pp. 1435 - 1469
• Main Authors: Mortier, Thomas, Wydmuch, Marek, Dembczyński, Krzysztof, Hüllermeier, Eyke, Waegeman, Willem
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.07.2021; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Chemistry and Earth Sciences; Classifiers; Computer Science; Data Mining and Knowledge Discovery; Decision theory; Expected utility; Information Storage and Retrieval; Physics; Special Issue of the Journal Track of ECML PKDD 2021; Statistics for Engineering; Uncertainty; Multi-class classification; Set-valued prediction; Expected utility maximization
• ISSN: 1384-5810; 1573-756X
• DOI: 10.1007/s10618-021-00751-x

In cases of uncertainty, a multi-class classifier preferably returns a set of candidate classes instead of predicting a single class label with little guarantee. More precisely, the classifier should strive for an optimal balance between the correctness (the true class is among the candidates) and the precision (the candidates are not too many) of its prediction. We formalize this problem within a general decision-theoretic framework that unifies most of the existing work in this area. In this framework, uncertainty is quantified in terms of conditional class probabilities, and the quality of a predicted set is measured in terms of a utility function. We then address the problem of finding the Bayes-optimal prediction, i.e., the subset of class labels with the highest expected utility. For this problem, which is computationally challenging as there are exponentially (in the number of classes) many predictions to choose from, we propose efficient algorithms that can be applied to a broad family of utility functions. Our theoretical results are complemented by experimental studies, in which we analyze the proposed algorithms in terms of predictive accuracy and runtime efficiency.