Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks

As Large Language Models (LLMs) continue to evolve, the search for efficient and meaningful evaluation methods is ongoing. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to int...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Zhao, Justin, Plaza-del-Arco, Flor Miriam, Genchel, Benjie, Curry, Amanda Cercas
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 21.10.2024
Subjects
Online AccessGet full text

Cover

Loading…
Abstract As Large Language Models (LLMs) continue to evolve, the search for efficient and meaningful evaluation methods is ongoing. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to intra-model bias, and many tasks - such as those related to emotional intelligence, creative writing, and persuasiveness - may be too subjective for a single model to judge fairly. We introduce the Language Model Council (LMC), where a group of LLMs collaborate to create tests, respond to them, and evaluate each other's responses to produce a ranking in a democratic fashion. Unlike previous approaches that focus on reducing cost or bias by using a panel of smaller models, our work examines the benefits and nuances of a fully inclusive LLM evaluation system. In a detailed case study on emotional intelligence, we deploy a council of 20 recent LLMs to rank each other on open-ended responses to interpersonal conflicts. Our results show that the LMC produces rankings that are more separable and more robust, and through a user study, we show that they are more consistent with human evaluations than any individual LLM judge. Using all LLMs for judging can be costly, however, so we use Monte Carlo simulations and hand-curated sub-councils to study hypothetical council compositions and discuss the value of the incremental LLM judge.
AbstractList As Large Language Models (LLMs) continue to evolve, the search for efficient and meaningful evaluation methods is ongoing. Many recent evaluations use LLMs as judges to score outputs from other LLMs, often relying on a single large model like GPT-4o. However, using a single LLM judge is prone to intra-model bias, and many tasks - such as those related to emotional intelligence, creative writing, and persuasiveness - may be too subjective for a single model to judge fairly. We introduce the Language Model Council (LMC), where a group of LLMs collaborate to create tests, respond to them, and evaluate each other's responses to produce a ranking in a democratic fashion. Unlike previous approaches that focus on reducing cost or bias by using a panel of smaller models, our work examines the benefits and nuances of a fully inclusive LLM evaluation system. In a detailed case study on emotional intelligence, we deploy a council of 20 recent LLMs to rank each other on open-ended responses to interpersonal conflicts. Our results show that the LMC produces rankings that are more separable and more robust, and through a user study, we show that they are more consistent with human evaluations than any individual LLM judge. Using all LLMs for judging can be costly, however, so we use Monte Carlo simulations and hand-curated sub-councils to study hypothetical council compositions and discuss the value of the incremental LLM judge.
Author Plaza-del-Arco, Flor Miriam
Genchel, Benjie
Zhao, Justin
Curry, Amanda Cercas
Author_xml – sequence: 1
  givenname: Justin
  surname: Zhao
  fullname: Zhao, Justin
– sequence: 2
  givenname: Flor
  surname: Plaza-del-Arco
  middlename: Miriam
  fullname: Plaza-del-Arco, Flor Miriam
– sequence: 3
  givenname: Benjie
  surname: Genchel
  fullname: Genchel, Benjie
– sequence: 4
  givenname: Amanda
  surname: Curry
  middlename: Cercas
  fullname: Curry, Amanda Cercas
BookMark eNqNi9EKgjAYRkcUZOU7DLoWbFOLLrPEi7rKe1vzT2dzK-eC3r5BPUBX34FzvhkaK61ghDxC6SrYRIRMkW9MG4YhSdYkjqmHLkemastqwCddgcSptooLucV76DTv2SA4k_KNd6B407H-LlSNMxdVTmn1fRnsKBd148KzvbbAB_ECXDBzNws0uTFpwP_tHC2zQ5HmwaPXTwtmKFtte-VUScNkQ6JVTAn9r_oAZEJGkA
ContentType Paper
Copyright 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID 8FE
8FG
ABJCF
ABUWG
AFKRA
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
HCIFZ
L6V
M7S
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
DatabaseName ProQuest SciTech Collection
ProQuest Technology Collection
Materials Science & Engineering Collection
ProQuest Central (Alumni)
ProQuest Central UK/Ireland
ProQuest Central Essentials
ProQuest Central
Technology Collection
ProQuest One Community College
ProQuest Central
SciTech Premium Collection (Proquest) (PQ_SDU_P3)
ProQuest Engineering Collection
Engineering Database
Publicly Available Content Database
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Academic
ProQuest One Academic UKI Edition
ProQuest Central China
Engineering Collection
DatabaseTitle Publicly Available Content Database
Engineering Database
Technology Collection
ProQuest Central Essentials
ProQuest One Academic Eastern Edition
ProQuest Central (Alumni Edition)
SciTech Premium Collection
ProQuest One Community College
ProQuest Technology Collection
ProQuest SciTech Collection
ProQuest Central China
ProQuest Central
ProQuest Engineering Collection
ProQuest One Academic UKI Edition
ProQuest Central Korea
Materials Science & Engineering Collection
ProQuest One Academic
Engineering Collection
DatabaseTitleList Publicly Available Content Database
Database_xml – sequence: 1
  dbid: 8FG
  name: ProQuest Technology Collection
  url: https://search.proquest.com/technologycollection1
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
Discipline Physics
EISSN 2331-8422
Genre Working Paper/Pre-Print
GroupedDBID 8FE
8FG
ABJCF
ABUWG
AFKRA
ALMA_UNASSIGNED_HOLDINGS
AZQEC
BENPR
BGLVJ
CCPQU
DWQXO
FRJ
HCIFZ
L6V
M7S
M~E
PIMPY
PQEST
PQQKQ
PQUKI
PRINS
PTHSS
ID FETCH-proquest_journals_30682415323
IEDL.DBID BENPR
IngestDate Thu Oct 24 04:33:34 EDT 2024
IsOpenAccess true
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-proquest_journals_30682415323
OpenAccessLink https://www.proquest.com/docview/3068241532?pq-origsite=%requestingapplication%
PQID 3068241532
PQPubID 2050157
ParticipantIDs proquest_journals_3068241532
PublicationCentury 2000
PublicationDate 20241021
PublicationDateYYYYMMDD 2024-10-21
PublicationDate_xml – month: 10
  year: 2024
  text: 20241021
  day: 21
PublicationDecade 2020
PublicationPlace Ithaca
PublicationPlace_xml – name: Ithaca
PublicationTitle arXiv.org
PublicationYear 2024
Publisher Cornell University Library, arXiv.org
Publisher_xml – name: Cornell University Library, arXiv.org
SSID ssj0002672553
Score 3.5695791
SecondaryResourceType preprint
Snippet As Large Language Models (LLMs) continue to evolve, the search for efficient and meaningful evaluation methods is ongoing. Many recent evaluations use LLMs as...
SourceID proquest
SourceType Aggregation Database
SubjectTerms Benchmarks
Councils
Emotional intelligence
Large language models
Robustness
Title Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks
URI https://www.proquest.com/docview/3068241532
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1bS8MwFD64FsE3r3iZI6CvwSzpbb4I09YhbgyZsLfZpileunYu88EXf7snXecehL2lpCnNafi-0y8nfACXCHmGhThlQgnquEzQTpa1qccEsmUc-Gll39YfeL1n52HsjmvBTddllStMrIA6LaXRyK8wtQ0M2wh-M_ukxjXK7K7WFhoNsDn-KTAL7G44GD79qSzc8zFnFv-AtmKPaBfsYTxT8z3YUsU-bFdFl1IfwMtjLRYS40iWE3M-XL7l1-ROTUtpPg0GMP8mXVxJr9O4UrXJ2gdpOUoTbJlqDbwRUeB9CWBkFOsPfQgXUTi67dHVa03qpaMn64mKI7CKslDHQIRylMzcII1Z4iQyiTGOCnOfjpdlPkuTE2huetLp5u4z2OHI1QaSebsJ1mL-pc6RaxdJCxpBdN-qw4pX_Z_wF-buiiA
link.rule.ids 783,787,12777,21400,33385,33756,43612,43817
linkProvider ProQuest
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1dT4MwFL3REaNvfsaPqU30tZFRPjZfTKZbUBlZDCZ7w1JKpjKY63zw33vLmHsw2VuTFtJemnMuh0sPwDVCnmYhi5pMMmo7JqOdLGtR12TIlrztpZV92yB0_Vf7aeSMasFN1WWVS0ysgDothdbIbzC1bWu2Ydbd9Itq1yj9dbW20NgEQx9VhS9fRrcXDl_-VBbL9TBnZv-AtmKP_i4YQz6Vsz3YkMU-bFVFl0IdwFtQi4VEO5LlRP8fLt7zW_IgJ6XQjwYDmP-QLu6k8YRXqjZZ-SAtrlIEW7paAwciCnwsAIxEXH2qQ7jq96J7ny6nFddbR8WrhbIjaBRlIY-BMGlLkTntlJuJnYiEYxwl5j4dN8s8M01OoLnuTqfruy9h248GQRw8hs9nsGMhb2t4tlpNaMxn3_IceXeeXNTB_QUU04sD
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Language+Model+Council%3A+Democratically+Benchmarking+Foundation+Models+on+Highly+Subjective+Tasks&rft.jtitle=arXiv.org&rft.au=Zhao%2C+Justin&rft.au=Plaza-del-Arco%2C+Flor+Miriam&rft.au=Genchel%2C+Benjie&rft.au=Curry%2C+Amanda+Cercas&rft.date=2024-10-21&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422