The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumpti...
Saved in:
Main Authors | , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
09.05.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Longstanding data labeling practices in machine learning involve collecting
and aggregating labels from multiple annotators. But what should we do when
annotators disagree? Though annotator disagreement has long been seen as a
problem to minimize, new perspectivist approaches challenge this assumption by
treating disagreement as a valuable source of information. In this position
paper, we examine practices and assumptions surrounding the causes of
disagreement--some challenged by perspectivist approaches, and some that remain
to be addressed--as well as practical and normative challenges for work
operating under these assumptions. We conclude with recommendations for the
data labeling pipeline and avenues for future research engaging with
subjectivity and disagreement. |
---|---|
DOI: | 10.48550/arxiv.2405.05860 |