Reinforcement-Learning-Based Robust Controller Design for Continuous-Time Uncertain Nonlinear Systems Subject to Input Constraints

• Published in: IEEE transactions on cybernetics Vol. 45; no. 7; pp. 1372 - 1385
• Main Authors: Liu, Derong, Yang, Xiong, Wang, Ding, Wei, Qinglai
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2015
• Subjects: Algorithm design and analysis; Approximate dynamic programming (ADP); Approximation algorithms; Artificial neural networks; neural networks (NNs); neuro-dynamic programming; Nonlinear systems; Optimal control; reinforcement learning (RL); Robust control; Robustness; nonlinear systems; neural networks (NNs); Approximate dynamic programming (ADP); optimal control; robust control; neuro-dynamic programming; reinforcement learning (RL)
• ISSN: 2168-2267; 2168-2275
• DOI: 10.1109/TCYB.2015.2417170

The design of stabilizing controller for uncertain nonlinear systems with control constraints is a challenging problem. The constrained-input coupled with the inability to identify accurately the uncertainties motivates the design of stabilizing controller based on reinforcement-learning (RL) methods. In this paper, a novel RL-based robust adaptive control algorithm is developed for a class of continuous-time uncertain nonlinear systems subject to input constraints. The robust control problem is converted to the constrained optimal control problem with appropriately selecting value functions for the nominal system. Distinct from typical action-critic dual networks employed in RL, only one critic neural network (NN) is constructed to derive the approximate optimal control. Meanwhile, unlike initial stabilizing control often indispensable in RL, there is no special requirement imposed on the initial control. By utilizing Lyapunov's direct method, the closed-loop optimal control system and the estimated weights of the critic NN are proved to be uniformly ultimately bounded. In addition, the derived approximate optimal control is verified to guarantee the uncertain nonlinear system to be stable in the sense of uniform ultimate boundedness. Two simulation examples are provided to illustrate the effectiveness and applicability of the present approach.