MARL-Based Multi-Satellite Intelligent Task Planning Method
In this article, we propose a solution to multi-satellite intelligent task planning using the multi-agent reinforcement learning (MARL) method. Fristly, we have developed a multi-satellite task planning model based on the Markov game framework. Furthermore, we have computationally designed a satelli...
Saved in:
| Published in | IEEE access Vol. 11; pp. 135517 - 135528 |
|---|---|
| Main Authors | , , , , , |
| Format | Journal Article |
| Language | English |
| Published |
Piscataway
IEEE
2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects | |
| Online Access | Get full text |
Cover
Loading…
| Abstract | In this article, we propose a solution to multi-satellite intelligent task planning using the multi-agent reinforcement learning (MARL) method. Fristly, we have developed a multi-satellite task planning model based on the Markov game framework. Furthermore, we have computationally designed a satellite state transition function to address the task planning problem and successfully solved it using the multi-agent proximal policy optimization (MAPPO) algorithm. Our experimental results demonstrate that the MARL method exhibits remarkable convergence speed and performance, delivering significant rewards in multi-scale task planning scenarios. Consequently, it proves to be a highly suitable approach for multi-satellite intelligent task planning. |
|---|---|
| AbstractList | In this article, we propose a solution to multi-satellite intelligent task planning using the multi-agent reinforcement learning (MARL) method. Fristly, we have developed a multi-satellite task planning model based on the Markov game framework. Furthermore, we have computationally designed a satellite state transition function to address the task planning problem and successfully solved it using the multi-agent proximal policy optimization (MAPPO) algorithm. Our experimental results demonstrate that the MARL method exhibits remarkable convergence speed and performance, delivering significant rewards in multi-scale task planning scenarios. Consequently, it proves to be a highly suitable approach for multi-satellite intelligent task planning. |
| Author | Zhang, Zhibin Li, Xinhong Zhang, Guohui Wang, Xun Hu, Gangxuan Li, Yanyan |
| Author_xml | – sequence: 1 givenname: Guohui orcidid: 0000-0002-3782-6696 surname: Zhang fullname: Zhang, Guohui organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China – sequence: 2 givenname: Xinhong surname: Li fullname: Li, Xinhong organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China – sequence: 3 givenname: Gangxuan surname: Hu fullname: Hu, Gangxuan organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China – sequence: 4 givenname: Yanyan surname: Li fullname: Li, Yanyan organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China – sequence: 5 givenname: Xun surname: Wang fullname: Wang, Xun organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China – sequence: 6 givenname: Zhibin orcidid: 0000-0002-8112-7227 surname: Zhang fullname: Zhang, Zhibin email: zhangzhibinseu@163.com organization: Department of Aerospace Science and Technology, Space Engineering University, Beijing, China |
| BookMark | eNpNUE1PwkAU3BhNROQX6KGJ5-Luvv3oxhM2qCQQjeB5s-2-YrG22JaD_95CieFd3mQyM-9lrsh5WZVIyA2jY8aouZ_E8XS5HHPKYQwAGmR0RgacKROCBHV-gi_JqGk2tJuoo6QekIfF5H0eProGfbDYFW0eLl2LRZG3GMzKA1pj2QYr13wFb4Ury7xcBwtsPyt_TS4yVzQ4Ou4h-XiaruKXcP76PIsn8zAV1LQhMiM5S9AbkXHBUw6JSJzyRqY6Ycx4L2imJYuM4EpTZAIznXnHXGIURAqGZNbn-spt7LbOv139ayuX2wNR1Wvr6jZPC7Te-SRCVB5Bigyoo-BRRkxLoeieHZK7PmtbVz87bFq7qXZ12b1veWSMUNpo2amgV6V11TQ1Zv9XGbX70m1fut2Xbo-ld67b3pUj4okDgHPJ4Q98H317 |
| CODEN | IAECCG |
| CitedBy_id | crossref_primary_10_3390_math12070986 |
| Cites_doi | 10.1109/TSMCB.2006.886173 10.1109/SSCI44817.2019.9002957 10.3390/rs13122377 10.1016/b978-1-55860-335-6.50027-1 10.1109/TGRS.2003.815999 10.1109/ACCESS.2018.2877687 10.1007/11536406_43 10.3390/aerospace9110676 10.1109/tfuzz.2023.3277480 10.1109/TSMC.2023.3277703 10.1016/j.cja.2018.12.018 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2023 |
| DBID | 97E ESBDL RIA RIE AAYXX CITATION 7SC 7SP 7SR 8BQ 8FD JG9 JQ2 L7M L~C L~D DOA |
| DOI | 10.1109/ACCESS.2023.3337358 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005-present IEEE Open Access Journals IEEE All-Society Periodicals Package (ASPP) 1998-Present IEEE Electronic Library Online CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Engineered Materials Abstracts METADEX Technology Research Database Materials Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional DOAJ Directory of Open Access Journals |
| DatabaseTitle | CrossRef Materials Research Database Engineered Materials Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace METADEX Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Materials Research Database |
| Database_xml | – sequence: 1 dbid: DOA name: DOAJ Directory of Open Access Journals url: https://www.doaj.org/ sourceTypes: Open Website – sequence: 2 dbid: RIE name: IEEE Electronic Library Online url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 2169-3536 |
| EndPage | 135528 |
| ExternalDocumentID | oai_doaj_org_article_dadb8ee6de354f30a03de581754606de 10_1109_ACCESS_2023_3337358 10332252 |
| Genre | orig-research |
| GroupedDBID | 0R~ 4.4 5VS 6IK 97E AAJGR ACGFS ADBBV ALMA_UNASSIGNED_HOLDINGS BCNDV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD ESBDL GROUPED_DOAJ IFIPE IPLJI JAVBF KQ8 M43 M~E O9- OCL OK1 RIA RIE RIG RNS AAYXX CITATION 7SC 7SP 7SR 8BQ 8FD JG9 JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c409t-e19521bed94f242c23b4ba6d95c7b119dd40f7518942670e14ef7fda1ab963863 |
| IEDL.DBID | DOA |
| ISSN | 2169-3536 |
| IngestDate | Tue Oct 22 15:14:53 EDT 2024 Thu Oct 10 17:40:54 EDT 2024 Fri Aug 23 01:01:47 EDT 2024 Mon Nov 04 11:48:22 EST 2024 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c409t-e19521bed94f242c23b4ba6d95c7b119dd40f7518942670e14ef7fda1ab963863 |
| ORCID | 0000-0002-3782-6696 0000-0002-8112-7227 |
| OpenAccessLink | https://doaj.org/article/dadb8ee6de354f30a03de581754606de |
| PQID | 2899467975 |
| PQPubID | 4845423 |
| PageCount | 12 |
| ParticipantIDs | crossref_primary_10_1109_ACCESS_2023_3337358 proquest_journals_2899467975 doaj_primary_oai_doaj_org_article_dadb8ee6de354f30a03de581754606de ieee_primary_10332252 |
| PublicationCentury | 2000 |
| PublicationDate | 20230000 2023-00-00 20230101 2023-01-01 |
| PublicationDateYYYYMMDD | 2023-01-01 |
| PublicationDate_xml | – year: 2023 text: 20230000 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE access |
| PublicationTitleAbbrev | Access |
| PublicationYear | 2023 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref12 ref14 Luo (ref11) 2021; 42 ref10 Du (ref15) 2019; 46 Sun (ref13) 2020; 56 ref19 ref18 Jun (ref1) 2007 He (ref2) ref7 Hoda (ref20) 2007 ref9 ref4 ref3 ref6 Liu (ref8) 2015; 36 ref5 Yu (ref21) 2021 Engstrom (ref22) 2020 Wen (ref16) 2023; 38 Luo (ref17) |
| References_xml | – year: 2021 ident: ref21 article-title: The surprising effectiveness of PPO in cooperative, multi-agent games publication-title: arXiv:2103.01955 contributor: fullname: Yu – ident: ref6 doi: 10.1109/TSMCB.2006.886173 – ident: ref9 doi: 10.1109/SSCI44817.2019.9002957 – ident: ref10 doi: 10.3390/rs13122377 – volume: 42 start-page: 524721 issue: 4 year: 2021 ident: ref11 article-title: Multi-satellite scheduling approach for emergency scenarios based on hierarchical forecasting with transformer network publication-title: Acta Aeronauticaet Astronautica Sinica contributor: fullname: Luo – volume-title: A gradient-based approach for computing Nash equilibria of large sequential games year: 2007 ident: ref20 contributor: fullname: Hoda – year: 2020 ident: ref22 article-title: Implementation matters in deep policy gradients: A case study on PPO and TRPO publication-title: arXiv:2005.12729 contributor: fullname: Engstrom – ident: ref14 doi: 10.1016/b978-1-55860-335-6.50027-1 – ident: ref3 doi: 10.1109/TGRS.2003.815999 – ident: ref18 doi: 10.1109/ACCESS.2018.2877687 – volume: 36 start-page: 583 issue: 5 year: 2015 ident: ref8 article-title: Schedulability prediction method for imaging tasks of earth observation network publication-title: J. Astronaut. contributor: fullname: Liu – ident: ref7 doi: 10.1007/11536406_43 – year: 2007 ident: ref1 article-title: An approach for multiobjective uniting imaging scheduling of earth observing satellites publication-title: J. Astronaut. contributor: fullname: Jun – volume: 38 start-page: 1200 issue: 5 year: 2023 ident: ref16 article-title: Reinforcement learning and adaptive/approximate dynamic programming: A survey from theory to applications in multi-agent systems publication-title: Control Decis. contributor: fullname: Wen – ident: ref12 doi: 10.3390/aerospace9110676 – volume-title: Proc. 4th Int. Conf. Syst. Sci. Syst. Eng. (ICSSSE) ident: ref2 article-title: Solving parallel machine scheduling problems with time windows using constraint programming and tabu search contributor: fullname: He – ident: ref4 doi: 10.1109/tfuzz.2023.3277480 – start-page: 1 ident: ref17 article-title: Study progress in multi-agent game learning publication-title: Syst. Eng. Elect. contributor: fullname: Luo – ident: ref5 doi: 10.1109/TSMC.2023.3277703 – volume: 46 start-page: 1 issue: 8 year: 2019 ident: ref15 article-title: Review of multi-agent reinforcement learning publication-title: Comput. Sci. contributor: fullname: Du – ident: ref19 doi: 10.1016/j.cja.2018.12.018 – volume: 56 start-page: 13 issue: 5 year: 2020 ident: ref13 article-title: Overview of multi-agent deep reinforcement learning publication-title: Comput. Eng. Appl. contributor: fullname: Sun |
| SSID | ssj0000816957 |
| Score | 2.3264415 |
| Snippet | In this article, we propose a solution to multi-satellite intelligent task planning using the multi-agent reinforcement learning (MARL) method. Fristly, we... |
| SourceID | doaj proquest crossref ieee |
| SourceType | Open Website Aggregation Database Publisher |
| StartPage | 135517 |
| SubjectTerms | Algorithms Games MAPPO Markov game Markov processes MARL multi-satellite intelligent task planning Multiagent systems Planning Predictive models Reinforcement learning Satellites Task analysis |
| SummonAdditionalLinks | – databaseName: IEEE Electronic Library Online dbid: RIE link: http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV09b9swECXSTOnQjyRF1DiFhoyhIoqUKCKTbTRwi9pD4gDZCFJ3WgI4RS0v_fU9UrJhpAiQTRAkiLo78vFI3nuMXQYGGEAnOLgauPKF4qb1gjuCykKDrF1ULZkvqtmD-vlYPg7F6rEWBhHj4TPMwmXcy4fnZhOWyqiHyxB_NOK-08b0xVq7BZWgIGFKPTALidxcj6dT-oksCIRnUkotg677HvpEkv5BVeW_oTjiy-1Htti2rD9W8pRtOp81f1-QNr656Z_Yh2GmmY770PjMDnB1zN7v8Q-esJv5-O4XnxCQQRorcfm9iwydHaY_dlydXbp066d0K2-UzqPo9Cl7uP2-nM74oKbAG8rhOo7CEFR7BKNawuWmkF55V4EpG-2FMAAqb8MmjCHQ1jkKha1uwQnnQyet5Bd2uHpe4RlLG180lFSjKjwq6WvjZAHkc115FCWohF1trWx_96QZNiYbubG9U2xwih2ckrBJ8MTu0cB4HW-QBe3QgSw48DViBShL1crc5RRjZU2zH0U5GGDCToPV977XGzxho61j7dA91zZkmYQQRpdfX3ntnB2FJvaLLSN22P3Z4AVNPzr_LYbdP60K1aQ priority: 102 providerName: IEEE |
| Title | MARL-Based Multi-Satellite Intelligent Task Planning Method |
| URI | https://ieeexplore.ieee.org/document/10332252 https://www.proquest.com/docview/2899467975 https://doaj.org/article/dadb8ee6de354f30a03de581754606de |
| Volume | 11 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV09T8MwELVQJxgQH0UUSpWBEbdx7MSxmNqKqiDKAK3UzbLjy4JUEA3_n7OTVpEYWFiTKI7fxfd8TvweIbdeAcaBYdSZ3FFhE0FVaRk1SJWJdDw3wbVk8ZLNV-Jpna5bVl_-n7BaHrgGbuSMszlA5oCnouSxifHeaY6sJ3Du7SBk31i1iqmQg3OWqVQ2MkN4fjSeTrFHQ-8WPuScS-5N3ltUFBT7G4uVX3k5kM3shBw3s8RoXD_dKTmAzRk5amkHnpP7xfj1mU6QhFwUdtHSNxPUNSuIHvc6m1W0NNv3aGdNFC2CYXSXrGYPy-mcNk4ItMD6q6LAFNKsBadEiZxaJNwKazKn0kJaxpRzIi79BxSFhCtjYAJKWTrDjPUDLOMXpLP52MAliQqbFFgQg0gsCG5zZXjiMF4ys8BSJ3rkbgeK_qwFL3QoFGKlawy1x1A3GPbIxAO3v9SrVYcDGEPdxFD_FcMe6XrYW-1xn2eSHunv4qCbobXVvkLE7K5kevUfbV-TQ9-felWlTzrV1zfc4DyjsoPwSg3ClsAfaXrNkQ |
| link.rule.ids | 315,783,787,799,867,2109,4031,27935,27936,27937,55086 |
| linkProvider | Directory of Open Access Journals |
| linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1Lb9QwEB6hcigcgJaibimQA8c6xLHzsDhtV1Rb2N1Du5V6s-zM5FJpW7XZC7--Yye7WoGQuEVRojjz8Ddje74B-BoYYJCcFOhqFNrnWpjWS-EYKvMKVe1i15L5opze6J-3xe1QrB5rYYgoHj6jNFzGvXy8b9ZhqYw9XAX74xn3JQfWddmXa22XVEIPCVNUA7eQzMy38WTCv5GGFuGpUqpSobP7Dv5Emv6hr8pfk3FEmIu3sNiMrT9YcpeuO582v_-gbfzvwb-DN0OsmYx74ziAF7Q6hNc7DITv4ft8fDUT5wxlmMRaXHHtIkdnR8nllq2zS5bu6S7ZNDhK5rHt9BHcXPxYTqZi6KcgGs7iOkHSMFh7QqNbRuYmV157V6IpmspLaRB11oZtGMOwXWUkNbVVi046H9y0VB9gb3W_omNIGp83nFaTzj1p5WvjVI6s9ar0JAvUIzjbSNk-9LQZNqYbmbG9UmxQih2UMoLzoInto4HzOt5gCdrBhSw69DVRiaQK3arMZWxlRc3xj-YsDGkER0HqO9_rBT6C041i7eCgTzbkmYwRpipO_vHaF9ifLuczO7tc_PoIr8Jw-6WXU9jrHtf0iYORzn-OJvgMAnbY7w |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MARL-Based+Multi-Satellite+Intelligent+Task+Planning+Method&rft.jtitle=IEEE+access&rft.au=Zhang%2C+Guohui&rft.au=Li%2C+Xinhong&rft.au=Hu%2C+Gangxuan&rft.au=Li%2C+Yanyan&rft.date=2023&rft.issn=2169-3536&rft.eissn=2169-3536&rft.volume=11&rft.spage=135517&rft.epage=135528&rft_id=info:doi/10.1109%2FACCESS.2023.3337358&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_ACCESS_2023_3337358 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2169-3536&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2169-3536&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2169-3536&client=summon |