Massively Parallel Methods for Deep Reinforcement Learning

• Main Authors: Nair, Arun, Srinivasan, Praveen, Blackwell, Sam, Alcicek, Cagdas, Fearon, Rory, De Maria, Alessandro, Panneershelvam, Vedavyas, Suleyman, Mustafa, Beattie, Charles, Petersen, Stig, Legg, Shane, Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David
• Format: Journal Article
• Language: English
• Published: 15.07.2015
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Distributed, Parallel, and Cluster Computing; Computer Science - Learning; Computer Science - Neural and Evolutionary Computing
• DOI: 10.48550/arxiv.1507.04296

We present the first massively distributed architecture for deep reinforcement learning. This architecture uses four main components: parallel actors that generate new behaviour; parallel learners that are trained from stored experience; a distributed neural network to represent the value function or behaviour policy; and a distributed store of experience. We used our architecture to implement the Deep Q-Network algorithm (DQN). Our distributed algorithm was applied to 49 games from Atari 2600 games from the Arcade Learning Environment, using identical hyperparameters. Our performance surpassed non-distributed DQN in 41 of the 49 games and also reduced the wall-time required to achieve these results by an order of magnitude on most games.