Structural relational inference actor-critic for multi-agent reinforcement learning

• Published in: Neurocomputing (Amsterdam) Vol. 459; pp. 383 - 394
• Main Authors: Zhang, Xianjie, Liu, Yu, Xu, Xiujuan, Huang, Qiong, Mao, Hangyu, Carie, Anil
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 12.10.2021
• Subjects: Actor-critic; Deep reinforcement learning; Graph neural network; Multi-agent systems; Variational autoencoder; Variational autoencoder; Deep reinforcement learning; Actor-critic; Graph neural network; Multi-agent systems
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2021.07.014

•A novel MARL algorithm involving the interaction relationship between agents.•This algorithm is based on centralized training and decentralized execution.•We apply the variational autoencoder model to obtain the latent relations.•We employ graph attention network to gather the information of neighbor agents. Multi-agent reinforcement learning (MARL) is essential for a wide range of high-dimensional scenarios and complicated tasks with multiple agents. Many attempts have been made for agents with prior domain knowledge and predefined structure. However, the interaction relationship between agents in a multi-agent system (MAS) in general is usually unknown, and previous methods could not tackle dynamical activities in an ever-changing environment. Here we propose a multi-agent Actor-Critic algorithm called Structural Relational Inference Actor-Critic (SRI-AC), which is based on the framework of centralized training and decentralized execution. SRI-AC utilizes the latent codes in variational autoencoder (VAE) to represent interactions between paired agents, and the reconstruction error is based on Graph Neural Network (GNN). With this framework, we test whether the reinforcement learning learners could form an interpretable structure while achieving better performance in both cooperative and competitive scenarios. The results indicate that SRI-AC could be applied to complex dynamic environments to find an interpretable structure while obtaining better performance compared to baseline algorithms.