Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models
Hate speech detection models are typically evaluated on held-out test sets. However, this risks painting an incomplete and potentially misleading picture of model performance because of increasingly well-documented systematic gaps and biases in hate speech datasets. To enable more targeted diagnosti...
Saved in:
Main Authors | , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
20.06.2022
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Hate speech detection models are typically evaluated on held-out test sets.
However, this risks painting an incomplete and potentially misleading picture
of model performance because of increasingly well-documented systematic gaps
and biases in hate speech datasets. To enable more targeted diagnostic
insights, recent research has thus introduced functional tests for hate speech
detection models. However, these tests currently only exist for
English-language content, which means that they cannot support the development
of more effective models in other languages spoken by billions across the
world. To help address this issue, we introduce Multilingual HateCheck (MHC), a
suite of functional tests for multilingual hate speech detection models. MHC
covers 34 functionalities across ten languages, which is more languages than
any other hate speech dataset. To illustrate MHC's utility, we train and test a
high-performing multilingual hate speech detection model, and reveal critical
model weaknesses for monolingual and cross-lingual applications. |
---|---|
DOI: | 10.48550/arxiv.2206.09917 |