Model-Based Reinforcement Learning for Infinite-Horizon Approximate Optimal Tracking

• Published in: IEEE transaction on neural networks and learning systems Vol. 28; no. 3; pp. 753 - 758
• Main Authors: Kamalapurkar, Rushikesh, Andrews, Lindsey, Walters, Patrick, Dixon, Warren E.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.03.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptive control; Aerospace electronics; Computer simulation; Convergence; Data-driven control; Learning; Machine learning; Markov analysis; Nonlinear control; Nonlinear systems; Optimal control; Reinforcement; reinforcement learning (RL); Stability analysis; Steady-state; system identification; Tracking control; Tracking problem; Trajectory
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2015.2511658

This brief paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. To relax the persistence of excitation condition, model-based reinforcement learning is implemented using a concurrent-learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.