EnDex: Evaluation of Dialogue Engagingness at Scale
We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and close...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
22.10.2022
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback. To support the soundness of our EnDex metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. We will release code, off-the-shelf EnDex model, and a large-scale dataset upon paper publication to facilitate future research. |
---|---|
AbstractList | We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated using a novel distant-supervision framework. Engagingness is a key measure that captures high-level quality of AI dialogue systems and closely reflects actual user experience. However, data shortage, plus the abstract and extensive definition of engagingness makes it challenging to develop an automatic metric. Our work departs from mainstream approaches that use synthetic negative examples to train binary classifiers, and instead, proposes a solution using distant-supervision from human-reaction feedback. To support the soundness of our EnDex metric, we offer a theoretical foundation for engagement, an extensive ablation study, and empirical evidence of high correlation on five engagingness related datasets. We will release code, off-the-shelf EnDex model, and a large-scale dataset upon paper publication to facilitate future research. |
Author | Xu, Guangxuan Liu, Ruibo Nischal Reddy Chandra Peng, Nanyun Harel-Canada, Fabrice |
Author_xml | – sequence: 1 givenname: Guangxuan surname: Xu fullname: Xu, Guangxuan – sequence: 2 givenname: Ruibo surname: Liu fullname: Liu, Ruibo – sequence: 3 givenname: Fabrice surname: Harel-Canada fullname: Harel-Canada, Fabrice – sequence: 4 fullname: Nischal Reddy Chandra – sequence: 5 givenname: Nanyun surname: Peng fullname: Peng, Nanyun |
BookMark | eNqNikEKwjAQAIMoWLV_CHgupBtri1cb8a73ssg2tISNNon4fHvwAV5mDjMbsWTPtBAZaF0WzQFgLfIQRqUUHGuoKp0Jbbilz0maN7qEcfAsfS_bAZ23iaRhi3ZgyxSCxChvD3S0E6seXaD8563YX8z9fC2ek38lCrEbfZp4Th3U0OiyKWf-d30B-CI1RQ |
ContentType | Paper |
Copyright | 2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection (Proquest) (PQ_SDU_P3) ProQuest Engineering Collection Engineering Database Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_27283181283 |
IEDL.DBID | BENPR |
IngestDate | Thu Oct 10 17:22:38 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_27283181283 |
OpenAccessLink | https://www.proquest.com/docview/2728318128?pq-origsite=%requestingapplication% |
PQID | 2728318128 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_2728318128 |
PublicationCentury | 2000 |
PublicationDate | 20221022 |
PublicationDateYYYYMMDD | 2022-10-22 |
PublicationDate_xml | – month: 10 year: 2022 text: 20221022 day: 22 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2022 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.4356933 |
SecondaryResourceType | preprint |
Snippet | We propose EnDex, the first human-reaction based model to evaluate dialogue engagingness. EnDex is trained on 80k Reddit-based Engagement Dataset (RED) curated... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Ablation Datasets Evaluation User experience |
Title | EnDex: Evaluation of Dialogue Engagingness at Scale |
URI | https://www.proquest.com/docview/2728318128 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LSwMxEB5sF8GbT3zUEtBrsM2myaaXgjZrEVqKD-it7CMRL9vHruDJ3-5kTetB6C0hkIQhfF_mm2EG4DbLu4xZ1aEKXR7KjVQ06VhLE2YkQgNXUV3xZjwRozf-NOvNvOBW-rTKDSbWQJ0vMqeR3zGJROjoKBosV9R1jXLRVd9CowEB63IXpg3u9WT6vFVZmJD4Zw7_AW3NHvEhBNNkadZHsGeKY9ivky6z8gRCXQzNV5_obcVtsrBk-PGrpxBdvNcthBwYkaQiL2hOcwo3sX59GNHNSXP_Gsr5393DM2iiW2_OgfA0tzKVticyy3GoRG4UM2HGLdK4yC-gtWuny93LV3DAXKI-oixjLWhW609zjfRZpW1oRPFj21sKZ-Nv_QNaDnnU |
link.rule.ids | 786,790,12792,21416,33406,33777,43633,43838 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LSwMxEB60RfTmEx9VA3oN1iRNNl482F1XbYtghd6W3TyKl23truDPN4nbehB6CwQyYRi-L_NlmAG4VvqWECu7WLqUBzMjJM671uKcGOGggckodLwZjnj6zp4nvUkjuFVNWeUSEwNQ65nyGvkNEY4IPR1F9_NP7KdG-d_VZoTGJrQZ5dTHeZQ8rjQWwoV7MdN_MBu4I9mF9ms-N4s92DDlPmyFkktVHQCNy775vkPxqt82mlnU__hVU1BcTsMAIQ9FKK_Rm3OmOYSrJB4_pHhpKWtiocr-bk6PoOWSenMMiBXaikLYHleWuaXk2khiqGLWkTjXJ9BZd9Lp-u1L2E7Hw0E2eBq9nMEO8SX7Dm8J6UCrXnyZc0ekdXERvPUDY4x5RA |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EnDex%3A+Evaluation+of+Dialogue+Engagingness+at+Scale&rft.jtitle=arXiv.org&rft.au=Xu%2C+Guangxuan&rft.au=Liu%2C+Ruibo&rft.au=Harel-Canada%2C+Fabrice&rft.au=Nischal+Reddy+Chandra&rft.date=2022-10-22&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |