Large-scale multi-agent reinforcement learning using image-based state representation

• Published in: 2016 IEEE 55th Conference on Decision and Control (CDC) pp. 7592 - 7597
• Main Authors: Tianshu Chu, Shuhui Qu, Jie Wang
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2016
• Subjects: Approximation algorithms; Correlation; Feature extraction; Kernel; Learning (artificial intelligence); Neural networks; Training
• DOI: 10.1109/CDC.2016.7799442

With its high-dimensional state and action space, large-scale multi-agent reinforcement learning (MARL) is a challenging problem. Centralized approximate RL is impractical to deal with this because the search cost grows exponentially with the number of agents. Further, traditional decentralized approaches require delicate model-specific decomposition and communication within multi-agent system (MAS). Recently, however, convolutional neural network (CNN) has been successfully integrated into perceptional RL with high-dimensional input images. If the information of a MAS can also be recorded by a stack of images, the spatial correlation would be naturally represented by `pixel' positions and the kernel filter that allows for the multi-scale communications in MAS. Based on this observation, this paper proposes a new image-based representation for MARL by combining CNN for feature extraction and linear regression for Q-function approximation. Additionally, we perform decentralized search for the optimal action based on the structure of the learning model, and then propose a distributed fitted Q-iteration framework for general-purpose MARL. Initial experiments demonstrate that this CNN-enhanced RL agent has competitive performance when compared to the traditional ones, which learn from a manually designed flat feature representation. These encouraging results demonstrate that the image-based representation can provide promising opportunities to represent and analyze large-scale MARL using deep learning techniques.