Robust Reinforcement Learning Control Using Integral Quadratic Constraints for Recurrent Neural Networks

• Published in: IEEE transactions on neural networks Vol. 18; no. 4; pp. 993 - 1002
• Main Authors: Anderson, C.W., Young, P.M., Buehner, M.R., Knight, J.N., Bush, K.A., Hittle, D.C.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.07.2007
• Subjects: Algorithms; Artificial Intelligence; Biomimetics - methods; Computer Simulation; Constraint theory; Decision Support Techniques; Feedback; Feedback control; Gain; Integral equations; Integral quadratic constraints (IQCs); Integrals; Learning; Least-Squares Analysis; Machine learning; Markov Chains; Models, Theoretical; Neural networks; Neural Networks (Computer); Recurrent neural networks; recurrent neural networks (NNs); Reinforcement; Reinforcement (Psychology); reinforcement learning; Robust control; Robust stability; Stability; Stability analysis; Uncertainty
• ISSN: 1045-9227; 1941-0093
• DOI: 10.1109/TNN.2007.899520

The applicability of machine learning techniques for feedback control systems is limited by a lack of stability guarantees. Robust control theory offers a framework for analyzing the stability of feedback control loops, but for the integral quadratic constraint (IQC) framework used here, all components are required to be represented as linear, time-invariant systems plus uncertainties with, for IQCs used here, bounded gain. In this paper, the stability of a control loop including a recurrent neural network (NN) is analyzed by replacing the nonlinear and time-varying components of the NN with IQCs on their gain. As a result, a range of the NN's weights is found within which stability is guaranteed. An algorithm is demonstrated for training the recurrent NN using reinforcement learning and guaranteeing stability while learning.