Evidence-based uncertainty sampling for active learning

• Published in: Data mining and knowledge discovery Vol. 31; no. 1; pp. 164 - 202
• Main Authors: Sharma, Manali, Bilgic, Mustafa
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.01.2017; Springer Nature B.V
• Subjects: Active learning; Artificial Intelligence; Chemistry and Earth Sciences; Computer Science; Data mining; Data Mining and Knowledge Discovery; Datasets; Efficiency; Empirical analysis; Heart; Image retrieval; Information Storage and Retrieval; Labeling; Learning; Mathematical analysis; Physics; Sampling; Statistics for Engineering; Strategy; Support vector machines; Teaching methods; Uncertainty; Active learning; Uncertainty sampling; Classification
• ISSN: 1384-5810; 1573-756X
• DOI: 10.1007/s10618-016-0460-3

Active learning methods select informative instances to effectively learn a suitable classifier. Uncertainty sampling, a frequently utilized active learning strategy, selects instances about which the model is uncertain but it does not consider the reasons for why the model is uncertain. In this article, we present an evidence-based framework that can uncover the reasons for why a model is uncertain on a given instance. Using the evidence-based framework, we discuss two reasons for uncertainty of a model: a model can be uncertain about an instance because it has strong, but conflicting evidence for both classes or it can be uncertain because it does not have enough evidence for either class. Our empirical evaluations on several real-world datasets show that distinguishing between these two types of uncertainties has a drastic impact on the learning efficiency. We further provide empirical and analytical justifications as to why distinguishing between the two uncertainties matters.