Active learning with biased non-response to label requests

• Published in: Data mining and knowledge discovery Vol. 38; no. 4; pp. 2117 - 2140
• Main Authors: Robinson, Thomas S., Tax, Niek, Mudd, Richard, Guy, Ido
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.07.2024; Springer Nature B.V
• Subjects: Algorithms; Annotations; Artificial Intelligence; Chemistry and Earth Sciences; Computer Science; Data Mining and Knowledge Discovery; Degradation; Expected utility; Information Storage and Retrieval; Labeling; Labels; Machine learning; Physics; Prediction models; Response bias; Sampling methods; Statistics for Engineering; CTR prediction; Missing data; E-commerce; Active learning; Non-response
• ISSN: 1384-5810; 1573-756X
• DOI: 10.1007/s10618-024-01026-x

Active learning can improve the efficiency of training prediction models by identifying the most informative new labels to acquire. However, non-response to label requests can impact active learning’s effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data, demonstrating that biased non-response is particularly detrimental to model performance. We argue that biased non-response is likely in contexts where the labelling process, by nature, relies on user interactions. To mitigate the impact of biased non-response, we propose a cost-based correction to the sampling strategy–the Upper Confidence Bound of the Expected Utility (UCB-EU) –that can, plausibly, be applied to any active learning algorithm. Through experiments, we demonstrate that our method successfully reduces the harm from labelling non-response in many settings. However, we also characterise settings where the non-response bias in the annotations remains detrimental under UCB-EU for specific sampling methods and data generating processes. Finally, we evaluate our method on a real-world dataset from an e-commerce platform. We show that UCB-EU yields substantial performance improvements to conversion models that are trained on clicked impressions. Most generally, this research serves to both better conceptualise the interplay between types of non-response and model improvements via active learning, and to provide a practical, easy-to-implement correction that mitigates model degradation.