Stream-based active learning for sliding windows under the influence of verification latency

• Published in: Machine learning Vol. 111; no. 6; pp. 2011 - 2036
• Main Authors: Pham, Tuan, Kottke, Daniel, Krempl, Georg, Sick, Bernhard
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.06.2022; Springer Nature B.V
• Subjects: Active learning; Artificial Intelligence; Classifiers; Computer Science; Control; Labels; Learning; Machine Learning; Mechatronics; Natural Language Processing (NLP); Robotics; Simulation; Simulation and Modeling; Sliding; Special Issue of the ECML PKDD 2022 Journal Track; Verification; Label delay; Verification latency; Evolving data streams; Active learning; Concept drift; Classification
• ISSN: 0885-6125; 1573-0565
• DOI: 10.1007/s10994-021-06099-z

Stream-based active learning (AL) strategies minimize the labeling effort by querying labels that improve the classifier’s performance the most. So far, these strategies neglect the fact that an oracle or expert requires time to provide a queried label. We show that existing AL methods deteriorate or even fail under the influence of such verification latency. The problem with these methods is that they estimate a label’s utility on the currently available labeled data. However, when this label would arrive, some of the current data may have gotten outdated and new labels have arrived. In this article, we propose to simulate the available data at the time when the label would arrive. Therefore, our method Forgetting and Simulating (FS) forgets outdated information and simulates the delayed labels to get more realistic utility estimates. We assume to know the label’s arrival date a priori and the classifier’s training data to be bounded by a sliding window. Our extensive experiments show that FS improves stream-based AL strategies in settings with both, constant and variable verification latency.