LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning
Developing interactive systems that leverage natural language instructions to solve complex robotic control tasks has been a long-desired goal in the robotics community. Large Language Models (LLMs) have demonstrated exceptional abilities in handling complex tasks, including logical reasoning, in-co...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
16.06.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Developing interactive systems that leverage natural language instructions to solve complex robotic control tasks has been a long-desired goal in the robotics community. Large Language Models (LLMs) have demonstrated exceptional abilities in handling complex tasks, including logical reasoning, in-context learning, and code generation. However, predicting low-level robotic actions using LLMs poses significant challenges. Additionally, the complexity of such tasks usually demands the acquisition of policies to execute diverse subtasks and combine them to attain the ultimate objective. Hierarchical Reinforcement Learning (HRL) is an elegant approach for solving such tasks, which provides the intuitive benefits of temporal abstraction and improved exploration. However, HRL faces the recurring issue of non-stationarity due to unstable lower primitive behaviour. In this work, we propose LGR2, a novel HRL framework that leverages language instructions to generate a stationary reward function for the higher-level policy. Since the language-guided reward is unaffected by the lower primitive behaviour, LGR2 mitigates non-stationarity and is thus an elegant method for leveraging language instructions to solve robotic control tasks. To analyze the efficacy of our approach, we perform empirical analysis and demonstrate that LGR2 effectively alleviates non-stationarity in HRL. Our approach attains success rates exceeding 70\(\%\) in challenging, sparse-reward robotic navigation and manipulation environments where the baselines fail to achieve any significant progress. Additionally, we conduct real-world robotic manipulation experiments and demonstrate that CRISP shows impressive generalization in real-world scenarios. |
---|---|
AbstractList | Developing interactive systems that leverage natural language instructions to solve complex robotic control tasks has been a long-desired goal in the robotics community. Large Language Models (LLMs) have demonstrated exceptional abilities in handling complex tasks, including logical reasoning, in-context learning, and code generation. However, predicting low-level robotic actions using LLMs poses significant challenges. Additionally, the complexity of such tasks usually demands the acquisition of policies to execute diverse subtasks and combine them to attain the ultimate objective. Hierarchical Reinforcement Learning (HRL) is an elegant approach for solving such tasks, which provides the intuitive benefits of temporal abstraction and improved exploration. However, HRL faces the recurring issue of non-stationarity due to unstable lower primitive behaviour. In this work, we propose LGR2, a novel HRL framework that leverages language instructions to generate a stationary reward function for the higher-level policy. Since the language-guided reward is unaffected by the lower primitive behaviour, LGR2 mitigates non-stationarity and is thus an elegant method for leveraging language instructions to solve robotic control tasks. To analyze the efficacy of our approach, we perform empirical analysis and demonstrate that LGR2 effectively alleviates non-stationarity in HRL. Our approach attains success rates exceeding 70\(\%\) in challenging, sparse-reward robotic navigation and manipulation environments where the baselines fail to achieve any significant progress. Additionally, we conduct real-world robotic manipulation experiments and demonstrate that CRISP shows impressive generalization in real-world scenarios. |
Author | Singh, Utsav Bhattacharyya, Pramit Namboodiri, Vinay P |
Author_xml | – sequence: 1 givenname: Utsav surname: Singh fullname: Singh, Utsav – sequence: 2 givenname: Pramit surname: Bhattacharyya fullname: Bhattacharyya, Pramit – sequence: 3 givenname: Vinay surname: Namboodiri middlename: P fullname: Namboodiri, Vinay P |
BookMark | eNqNyksKwjAYBOAgClbtHQKuCzGxD92JaF10VVwKJaZ_a0pMNG3w-qbgAVzNMPMt0FQbDRMUUMY2UbaldI7Cvu8IITRJaRyzAN2KvKR7XHDdOt4Czp2socYlfLgdQ_E7KKlb3BiLD0KAAsuHcbhI36x4SMGVh1J7IeAJesAFcKu9WaFZw1UP4S-XaH0-XY-X6GXN20E_VJ1xVvurYiRJ4jTOdpT9p76t1ESP |
ContentType | Paper |
Copyright | 2024. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2024. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials AUTh Library subscriptions: ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection (Proquest) (PQ_SDU_P3) ProQuest Engineering Collection Engineering Database Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_30665758923 |
IEDL.DBID | BENPR |
IngestDate | Tue Sep 24 23:10:17 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_30665758923 |
OpenAccessLink | https://www.proquest.com/docview/3066575892/abstract/?pq-origsite=%requestingapplication% |
PQID | 3066575892 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_3066575892 |
PublicationCentury | 2000 |
PublicationDate | 20240616 |
PublicationDateYYYYMMDD | 2024-06-16 |
PublicationDate_xml | – month: 06 year: 2024 text: 20240616 day: 16 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2024 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.5412178 |
SecondaryResourceType | preprint |
Snippet | Developing interactive systems that leverage natural language instructions to solve complex robotic control tasks has been a long-desired goal in the robotics... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Cognition & reasoning Control tasks Empirical analysis Interactive control Interactive systems Language Large language models Robot control Robot learning Robotics Task complexity |
Title | LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning |
URI | https://www.proquest.com/docview/3066575892/abstract/ |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LS8NAEB7aBsGbT3zUsqDXkGZ3myZeRCUPpC0lKPQglGSzK4WS1ja9-tvdWRM9CL3ti2VfzCzffDMDcMd8qZQbCNvPlLD5UFI7ywthswyD3RTKzyk6J48nXvLGX2aDWQuSxhcGaZWNTDSCulgJxMgd9mMj8APqZDmiAKJyHtafNuaPQjtrnUyjDRZ1ORpsradwMk1_8RbqDfXvmf0TuUaPREdgTbO13BxDS5YncGDol2J7Cu-jOKX3ZFSDhyTeLQpZkFQip5UgYS03fuNEfzHJoxBaV-DN6YZkgR7EJqHJUg80cVCFgfxIHTr14wxuo_D1ObGbJc3rB7Sd_22XnUOnXJXyAog-Pj7giilfcV1CUiBXWb8QigeC9fkldPfNdLW_-xoOqdbYyINyvS50qs1O3miNW-U9aPtR3KuPVNfGX-E3_mKM1w |
link.rule.ids | 786,790,12792,21416,33408,33779,43635,43840 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LSwMxEB60RfTmEx9VA3oNtkm63fUiIu6uui1SKvQgLNk8pCBau-3_NxNTPQi9hWQJSTbMl3z5ZgbgksfG2k6iaCytoqJnGJWVVpRLDHajbVwxdE7uD6L8RTyOu-NAuNVBVrm0id5Q60-FHPkV_3kjiBN2M_2imDUKX1dDCo11aAruoBM9xdPsl2NhUc-dmPk_M-uxI92G5rOcmtkOrJmPXdjwkktV78FrkQ3ZNSkCYUiyxUQbTYYGdawERWqV9xUn7lhJbpVy-IB_y1XkE_Qa9klM3t2HPvap8jQfCeFS3_bhIr0f3eV0OaQybJq6_JsiP4CGu_2bQyBuyURXWG5jK1wJhYDCyrZWViSKt8URtFb1dLy6-Rw281G_KIuHwdMJbDGH2KiD6kQtaMxnC3PqEHdenfll_QYQQYiN |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LGR2%3A+Language+Guided+Reward+Relabeling+for+Accelerating+Hierarchical+Reinforcement+Learning&rft.jtitle=arXiv.org&rft.au=Singh%2C+Utsav&rft.au=Bhattacharyya%2C+Pramit&rft.au=Namboodiri%2C+Vinay+P&rft.date=2024-06-16&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |