Learning for multi-robot cooperation in partially observable stochastic environments with macro-actions

• Published in: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) pp. 1853 - 1860
• Main Authors: Miao Liu, Sivakumar, Kavinayan, Omidshafiei, Shayegan, Amato, Christopher, How, Jonathan P.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.09.2017
• Subjects: Decision making; History; Manganese; Power capacitors; Robot sensing systems; Stochastic processes
• ISSN: 2153-0866
• DOI: 10.1109/IROS.2017.8206001

This paper presents a data-driven approach for multi-robot coordination in partially-observable domains based on Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) and macro-actions (MAs). Dec-POMDPs provide a general framework for cooperative sequential decision making under uncertainty and MAs allow temporally extended and asynchronous action execution. To date, most methods assume the underlying Dec-POMDP model is known a priori or a full simulator is available during planning time. Previous methods which aim to address these issues suffer from local optimality and sensitivity to initial conditions. Additionally, few hardware demonstrations involving a large team of heterogeneous robots and with long planning horizons exist. This work addresses these gaps by proposing an iterative sampling based Expectation-Maximization algorithm (iSEM) to learn polices using only trajectory data containing observations, MAs, and rewards. Our experiments show the algorithm is able to achieve better solution quality than the state-of-the-art learning-based methods. We implement two variants of multi-robot Search and Rescue (SAR) domains (with and without obstacles) on hardware to demonstrate the learned policies can effectively control a team of distributed robots to cooperate in a partially observable stochastic environment.