Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback w...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , , , , , , , , , , , , , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
17.10.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild. |
---|---|
AbstractList | Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild. |
Author | Liang, Youwei Singla, Sahil Hu, Yushi Li, Gang Bitton, Yonatan Ramachandran, Deepak He, Junfeng Krishnamurthy Dj Dvijotham Kim, Najoung Chen, Sherol Chang, Minsuk Collins, Katherine M Evans, Georgina Rieser, Verena Lee, Kimin Weller, Adrian Shayegan Omidshafiei Dutta, Senjuti |
Author_xml | – sequence: 1 givenname: Katherine surname: Collins middlename: M fullname: Collins, Katherine M – sequence: 2 givenname: Najoung surname: Kim fullname: Kim, Najoung – sequence: 3 givenname: Yonatan surname: Bitton fullname: Bitton, Yonatan – sequence: 4 givenname: Verena surname: Rieser fullname: Rieser, Verena – sequence: 5 fullname: Shayegan Omidshafiei – sequence: 6 givenname: Yushi surname: Hu fullname: Hu, Yushi – sequence: 7 givenname: Sherol surname: Chen fullname: Chen, Sherol – sequence: 8 givenname: Senjuti surname: Dutta fullname: Dutta, Senjuti – sequence: 9 givenname: Minsuk surname: Chang fullname: Chang, Minsuk – sequence: 10 givenname: Kimin surname: Lee fullname: Lee, Kimin – sequence: 11 givenname: Youwei surname: Liang fullname: Liang, Youwei – sequence: 12 givenname: Georgina surname: Evans fullname: Evans, Georgina – sequence: 13 givenname: Sahil surname: Singla fullname: Singla, Sahil – sequence: 14 givenname: Gang surname: Li fullname: Li, Gang – sequence: 15 givenname: Adrian surname: Weller fullname: Weller, Adrian – sequence: 16 givenname: Junfeng surname: He fullname: He, Junfeng – sequence: 17 givenname: Deepak surname: Ramachandran fullname: Ramachandran, Deepak – sequence: 18 fullname: Krishnamurthy Dj Dvijotham |
BookMark | eNqNyk1uwjAQQGGrAqlQcoeRWFsYm5SoywIB9skaOc3kh4YZsB0Bt4cFB2D1Ld4biwEx4YcYaWPmMllo_Ski749KKf291HFsRqL4xTtTCVnTnwoP-Xm25iv9QE7BUt21VMOqsV2HVKMHriBtCeXW2SclpIhlYf_-oWIHGd6CDCz3J1sjbJHQ2dAyTcSwsp3H6OWXmKabbLWTZ8eXHn04HLl39EwHo5ZaxYlaJOa96wFhdUYx |
ContentType | Paper |
Copyright | 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central Korea SciTech Premium Collection ProQuest Engineering Collection Engineering Database Publicly Available Content (ProQuest) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_30720580483 |
IEDL.DBID | 8FG |
IngestDate | Sat Oct 19 04:28:18 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_30720580483 |
OpenAccessLink | https://www.proquest.com/docview/3072058048?pq-origsite=%requestingapplication% |
PQID | 3072058048 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_3072058048 |
PublicationCentury | 2000 |
PublicationDate | 20241017 |
PublicationDateYYYYMMDD | 2024-10-17 |
PublicationDate_xml | – month: 10 year: 2024 text: 20241017 day: 17 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2024 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.571424 |
SecondaryResourceType | preprint |
Snippet | Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Alignment Feedback Image processing Image quality Learning |
Title | Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation |
URI | https://www.proquest.com/docview/3072058048 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1Na4NAEB3aSKG3ftKmaVhorxJdza7ppdBUkxYSQlHILazr2kKJ2miu_e2dNdoeCgEvIogus--9GWd8APep56aMSWpSyjFBUU6KW0paphXb6WgoRMKlHk6ezdk0cl-Xw2VTcCubtsoWE2ugTnKpa-QDjEVqDT0MuMfiy9SuUfrramOhcQiGTTnTLX1eMPmtsVDGUTE7_2C25o7gBIyFKNTmFA5UdgZHdculLM8h3g2PkPBju45LEhWDZ8yIH0iUoV5711PiZNw6nZQkT0mAgtCcaEsHlZAAWScW8pOg6CShTl-r3HxZIzqQ3Z-k9YJfwF3gh-Op2T7YqgmdcvX3os4ldLI8U1d6slrgkXiuQMEgFXK0TVmitGsGE9aIX0Nv3526-y_fwDFFrtaQbPMedKrNVt0i11Zxv17QPhhP_nzxhmezb_8HHvyJhw |
link.rule.ids | 780,784,12765,21388,33373,33744,43600,43805 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1NT4NAFHzRNkZvfsaPqpvodVO6wEK9eKhSqm3jAZLeyLIsNjEFLPT_-5aCHkyacCMhsHnMzHvsMACPqWulnEtGGXOwQVFmiq-UNKgRD9KhLUTiSG1Ons25H1pvC3vRDNzKZltli4k1UCe51DPyPtYiM2wXC-65-KY6NUp_XW0iNPaha5lI3dop7o1_ZyyMO6iYzX8wW3OHdwzdD1Go9QnsqewUDuotl7I8g3hrHiHBcrOKSxIW_RfsiJ9ImKFe-9QucTJqk05KkqfEQ0FIxzrSQSXEQ9aJhfwiKDpJoNvXKqeTFaID2f5JWi_4OTx4r8HIp-2NRU3plNHfg5oX0MnyTF1qZ7XAI3EtgYJBKuToAeOJ0qkZXBhD5wp6u650vfv0PRz6wWwaTSfz9xs4YsjbGp4HTg861XqjbpF3q_iuXtwfFmqJng |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Beyond+Thumbs+Up%2FDown%3A+Untangling+Challenges+of+Fine-Grained+Feedback+for+Text-to-Image+Generation&rft.jtitle=arXiv.org&rft.au=Collins%2C+Katherine+M&rft.au=Kim%2C+Najoung&rft.au=Bitton%2C+Yonatan&rft.au=Rieser%2C+Verena&rft.date=2024-10-17&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |