Discrete-Time Deterministic Q -Learning: A Novel Convergence Analysis

• Published in: IEEE transactions on cybernetics Vol. 47; no. 5; pp. 1224 - 1237
• Main Authors: Qinglai Wei, Lewis, Frank L., Qiuye Sun, Pengfei Yan, Ruizhuo Song
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.05.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: <italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <inline-formula> <tex-math notation="LaTeX"> Q </tex-math> </inline-formula> -learning; Acquisitions & mergers; Adaptive critic designs; adaptive dynamic programming (ADP); Algorithm design and analysis; Algorithms; approximate dynamic programming; Computer simulation; Convergence; Criteria; Discrete-time systems; Dynamic programming; Iterative algorithms; Iterative methods; Lower bounds; Machine learning; Neural networks; neural networks (NNs); neuro-dynamic programming; Optimal control
• ISSN: 2168-2267; 2168-2275
• DOI: 10.1109/TCYB.2016.2542923

In this paper, a novel discrete-time deterministic Q-learning algorithm is developed. In each iteration of the developed Q-learning algorithm, the iterative Q function is updated for all the state and control spaces, instead of updating for a single state and a single control in traditional Q-learning algorithm. A new convergence criterion is established to guarantee that the iterative Q function converges to the optimum, where the convergence criterion of the learning rates for traditional Q-learning algorithms is simplified. During the convergence analysis, the upper and lower bounds of the iterative Q function are analyzed to obtain the convergence criterion, instead of analyzing the iterative Q function itself. For convenience of analysis, the convergence properties for undiscounted case of the deterministic Q-learning algorithm are first developed. Then, considering the discounted factor, the convergence criterion for the discounted case is established. Neural networks are used to approximate the iterative Q function and compute the iterative control law, respectively, for facilitating the implementation of the deterministic Q-learning algorithm. Finally, simulation results and comparisons are given to illustrate the performance of the developed algorithm.