ECoh: Turn-level Coherence Evaluation for Multilingual Dialogues
Despite being heralded as the new standard for dialogue evaluation, the closed-source nature of GPT-4 poses challenges for the community. Motivated by the need for lightweight, open source, and multilingual dialogue evaluators, this paper introduces GenResCoh (Generated Responses targeting Coherence...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
16.07.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Despite being heralded as the new standard for dialogue evaluation, the
closed-source nature of GPT-4 poses challenges for the community. Motivated by
the need for lightweight, open source, and multilingual dialogue evaluators,
this paper introduces GenResCoh (Generated Responses targeting Coherence).
GenResCoh is a novel LLM generated dataset comprising over 130k negative and
positive responses and accompanying explanations seeded from XDailyDialog and
XPersona covering English, French, German, Italian, and Chinese. Leveraging
GenResCoh, we propose ECoh (Evaluation of Coherence), a family of evaluators
trained to assess response coherence across multiple languages. Experimental
results demonstrate that ECoh achieves multilingual detection capabilities
superior to the teacher model (GPT-3.5-Turbo) on GenResCoh, despite being based
on a much smaller architecture. Furthermore, the explanations provided by ECoh
closely align in terms of quality with those generated by the teacher model. |
---|---|
DOI: | 10.48550/arxiv.2407.11660 |