Optimal and Autonomous Control Using Reinforcement Learning: A Survey

• Published in: IEEE transaction on neural networks and learning systems Vol. 29; no. 6; pp. 2042 - 2062
• Main Authors: Kiumarsi, Bahare, Vamvoudakis, Kyriakos G., Modares, Hamidreza, Lewis, Frank L.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.06.2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithms; Approximation algorithms; Autonomy; Computer & video games; data-based optimization; Feedback control; Games; H-infinity control; Heuristic algorithms; Learning; Learning (artificial intelligence); Machine learning; Multiagent systems; Optimal control; Reinforcement; reinforcement learning (RL); State-of-the-art reviews; System dynamics
• DOI: 10.1109/TNNLS.2017.2773458
• ISSN: 2162-237X; 2162-2388

This paper reviews the current state of the art on reinforcement learning (RL)-based feedback control solutions to optimal regulation and tracking of single and multiagent systems. Existing RL solutions to both optimal <inline-formula> <tex-math notation="LaTeX">\mathcal {H}_{2} </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">\mathcal {H}_\infty </tex-math></inline-formula> control problems, as well as graphical games, will be reviewed. RL methods learn the solution to optimal control and game problems online and using measured data along the system trajectories. We discuss Q-learning and the integral RL algorithm as core algorithms for discrete-time (DT) and continuous-time (CT) systems, respectively. Moreover, we discuss a new direction of off-policy RL for both CT and DT systems. Finally, we review several applications.