ML-EAT: A Multilevel Embedding Association Test for Interpretable and Transparent Social Science
This research introduces the Multilevel Embedding Association Test (ML-EAT), a method designed for interpretable and transparent measurement of intrinsic bias in language technologies. The ML-EAT addresses issues of ambiguity and difficulty in interpreting the traditional EAT measurement by quantify...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
04.08.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | This research introduces the Multilevel Embedding Association Test (ML-EAT),
a method designed for interpretable and transparent measurement of intrinsic
bias in language technologies. The ML-EAT addresses issues of ambiguity and
difficulty in interpreting the traditional EAT measurement by quantifying bias
at three levels of increasing granularity: the differential association between
two target concepts with two attribute concepts; the individual effect size of
each target concept with two attribute concepts; and the association between
each individual target concept and each individual attribute concept. Using the
ML-EAT, this research defines a taxonomy of EAT patterns describing the nine
possible outcomes of an embedding association test, each of which is associated
with a unique EAT-Map, a novel four-quadrant visualization for interpreting the
ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2
language models, and a CLIP language-and-image model shows that EAT patterns
add otherwise unobservable information about the component biases that make up
an EAT; reveal the effects of prompting in zero-shot models; and can also
identify situations when cosine similarity is an ineffective metric, rendering
an EAT unreliable. Our work contributes a method for rendering bias more
observable and interpretable, improving the transparency of computational
investigations into human minds and societies. |
---|---|
DOI: | 10.48550/arxiv.2408.01966 |