Needle in a Haystack: Label-Efficient Evaluation under Extreme Class Imbalance

Important tasks like record linkage and extreme classification demonstrate extreme class imbalance, with 1 minority instance to every 1 million or more majority instances. Obtaining a sufficient sample of all classes, even just to achieve statistically-significant evaluation, is so challenging that...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Marchant, Neil G, Rubinstein, Benjamin I P
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 02.06.2021
Subjects
Adaptive sampling
Asymptotic properties
Business metrics
Confidence intervals
Dirichlet problem
Importance sampling
Markov analysis
Martingales
Performance evaluation
Samplers
Online AccessGet full text

Cover

More Information
Summary:Important tasks like record linkage and extreme classification demonstrate extreme class imbalance, with 1 minority instance to every 1 million or more majority instances. Obtaining a sufficient sample of all classes, even just to achieve statistically-significant evaluation, is so challenging that most current approaches yield poor estimates or incur impractical cost. Where importance sampling has been levied against this challenge, restrictive constraints are placed on performance metrics, estimates do not come with appropriate guarantees, or evaluations cannot adapt to incoming labels. This paper develops a framework for online evaluation based on adaptive importance sampling. Given a target performance metric and model for \(p(y|x)\), the framework adapts a distribution over items to label in order to maximize statistical precision. We establish strong consistency and a central limit theorem for the resulting performance estimates, and instantiate our framework with worked examples that leverage Dirichlet-tree models. Experiments demonstrate an average MSE superior to state-of-the-art on fixed label budgets.
ISSN:2331-8422