Do Deep Reinforcement Learning Agents Model Intentions?

• Published in: Stats (Basel, Switzerland) Vol. 6; no. 1; pp. 50 - 66
• Main Authors: Matiisen, Tambet, Labash, Aqeel, Majoral, Daniel, Aru, Jaan, Vicente, Raul
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.03.2023
• Subjects: Adaptation; Algorithms; Analysis; artificial neural networks; Autonomous vehicles; Behavior; Deep learning; multi-agent reinforcement learning; Multi-agent systems; Neural networks; Reinforcement learning (Machine learning); Theory of mind; Estonia
• ISSN: 2571-905X; 2571-905X
• DOI: 10.3390/stats6010004

Inferring other agents’ mental states, such as their knowledge, beliefs and intentions, is thought to be essential for effective interactions with other agents. Recently, multi-agent systems trained via deep reinforcement learning have been shown to succeed in solving various tasks. Still, how each agent models or represents other agents in their environment remains unclear. In this work, we test whether deep reinforcement learning agents trained with the multi-agent deep deterministic policy gradient (MADDPG) algorithm explicitly represent other agents’ intentions (their specific aims or plans) during a task in which the agents have to coordinate the covering of different spots in a 2D environment. In particular, we tracked over time the performance of a linear decoder trained to predict the final targets of all agents from the hidden-layer activations of each agent’s neural network controller. We observed that the hidden layers of agents represented explicit information about other agents’ intentions, i.e., the target landmark the other agent ended up covering. We also performed a series of experiments in which some agents were replaced by others with fixed targets to test the levels of generalization of the trained agents. We noticed that during the training phase, the agents developed a preference for each landmark, which hindered generalization. To alleviate the above problem, we evaluated simple changes to the MADDPG training algorithm which lead to better generalization against unseen agents. Our method for confirming intention modeling in deep learning agents is simple to implement and can be used to improve the generalization of multi-agent systems in fields such as robotics, autonomous vehicles and smart cities.