Challenges of real-world reinforcement learning: definitions, benchmarks and analysis

• Published in: Machine learning Vol. 110; no. 9; pp. 2419 - 2468
• Main Authors: Dulac-Arnold, Gabriel, Levine, Nir, Mankowitz, Daniel J., Li, Jerry, Paduraru, Cosmin, Gowal, Sven, Hester, Todd
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.09.2021; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Benchmarks; Computer Science; Control; Decision analysis; Electronic health records; Machine Learning; Markov processes; Mechatronics; Natural Language Processing (NLP); Recommender systems; Robotics; Simulation and Modeling; Software; Special Issue on Reinforcement Learning for Real Life; Applied reinforcement learning; Reinforcement learning; Real-world
• ISSN: 0885-6125; 1573-0565
• DOI: 10.1007/s10994-021-05961-4

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. In this work, we identify and formalize a series of independent challenges that embody the difficulties that must be addressed for RL to be commonly deployed in real-world systems. For each challenge, we define it formally in the context of a Markov Decision Process, analyze the effects of the challenge on state-of-the-art learning algorithms, and present some existing attempts at tackling it. We believe that an approach that addresses our set of proposed challenges would be readily deployable in a large number of real world problems. Our proposed challenges are implemented in a suite of continuous control environments called realworldrl-suite which we propose an as an open-source benchmark.