Zero-failure testing of binary classifiers
We propose using performance metrics derived from zero-failure testing to assess binary classifiers. The principal characteristic of the proposed approach is the asymmetric treatment of the two types of error. In particular, we construct a test set consisting of positive and negative samples, set th...
Saved in:
Main Authors | , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
04.07.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | We propose using performance metrics derived from zero-failure testing to
assess binary classifiers. The principal characteristic of the proposed
approach is the asymmetric treatment of the two types of error. In particular,
we construct a test set consisting of positive and negative samples, set the
operating point of the binary classifier at the lowest value that will result
to correct classifications of all positive samples, and use the algorithm's
success rate on the negative samples as a performance measure. A property of
the proposed approach, setting it apart from other commonly used testing
methods, is that it allows the construction of a series of tests of increasing
difficulty, corresponding to a nested sequence of positive sample test sets. We
illustrate the proposed method on the problem of age estimation for determining
whether a subject is above a legal age threshold, a problem that exemplifies
the asymmetry of the two types of error. Indeed, misclassifying an under-aged
subject is a legal and regulatory issue, while misclassifications of people
above the legal age is an efficiency issue primarily concerning the commercial
user of the age estimation system. |
---|---|
DOI: | 10.48550/arxiv.2407.03979 |