A Hetero-Relation Transformer Network for Multi-Agent Reinforcement Learning

• Published in: IEEE transactions on games pp. 1 - 11
• Main Authors: Park, Junho, Yoon, Sukmin, Kim, Yong-Duk
• Format: Journal Article
• Language: English
• Published: IEEE 2024
• Subjects: Heterogeneous agent; multi-agent reinforcement learning; transformer
• ISSN: 2475-1502; 2475-1510
• DOI: 10.1109/TG.2024.3399167

Recently, considerable research has been focused on multi-agent reinforcement learning to effectively account for each agent's relations. However, most research has focused on homogeneous multi-agent systems with the same type of agents, which has limited application to heterogeneous multi-agent systems. The demand for heterogeneous systems has considerably increased not only in games but also in the real world. Therefore, a technique that can properly consider relations in heterogeneous systems is required. In this study, we propose a novel transformer network called HRformer , which is based on heterogeneous graph networks that can reflect the heterogeneity and relations among agents. To this end, we design an effective linear encoding method for the transformer to receive input of the various and unique characteristics of the agents and introduce a novel encoding method to model the relations among them. Experiments are conducted in the StarCraft Multi-Agent Challenge (SMAC) environment, the most famous heterogeneous multi-agent simulation, to demonstrate the superior performance of the proposed method compared to other existing methods in various heterogeneous scenarios. The proposed method in our simulation shows a high win rate and fast convergence speed, proving the superiority of the proposed method considering the heterogeneity of the multi-agent system.