Who Reviews The Reviewers? A Multi-Level Jury Problem
We consider the problem of determining a binary ground truth using advice from a group of independent reviewers (experts) who express their guess about a ground truth correctly with some independent probability (competence). In this setting, when all reviewers are competent (competence greater than...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
15.11.2022
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | We consider the problem of determining a binary ground truth using advice
from a group of independent reviewers (experts) who express their guess about a
ground truth correctly with some independent probability (competence). In this
setting, when all reviewers are competent (competence greater than one-half),
the Condorcet Jury Theorem tells us that adding more reviewers increases the
overall accuracy, and if all competences are known, then there exists an
optimal weighting of the reviewers. However, in practical settings, reviewers
may be noisy or incompetent, i.e., competence below half, and the number of
experts may be small, so the asymptotic Condorcet Jury Theorem is not
practically relevant. In such cases we explore appointing one or more chairs
(judges) who determine the weight of each reviewer for aggregation, creating
multiple levels. However, these chairs may be unable to correctly identify the
competence of the reviewers they oversee, and therefore unable to compute the
optimal weighting. We give conditions when a set of chairs is able to weight
the reviewers optimally, and depending on the competence distribution of the
agents, give results about when it is better to have more chairs or more
reviewers. Through numerical simulations we show that in some cases it is
better to have more chairs, but in many cases it is better to have more
reviewers. |
---|---|
DOI: | 10.48550/arxiv.2211.08494 |