Linear stochastic approximation driven by slowly varying Markov chains

We study a linear stochastic approximation algorithm that arises in the context of reinforcement learning. The algorithm employs a decreasing step-size, and is driven by Markov noise with time-varying statistics. We show that under suitable conditions, the algorithm can track the changes in the stat...

Full description

Saved in:
Bibliographic Details
Published inSystems & control letters Vol. 50; no. 2; pp. 95 - 102
Main Authors Konda, Vijay R., Tsitsiklis, John N.
Format Journal Article
LanguageEnglish
Published Amsterdam Elsevier B.V 07.10.2003
Elsevier
Subjects
Adaptive algorithms
Applied sciences
Computer science; control theory; systems
Control system analysis
Control theory. Systems
Exact sciences and technology
Modelling and identification
Reinforcement learning
Stochastic approximation
Stochastic approximation
Adaptive algorithms
Reinforcement learning
Markov chain
Time varying system
Adaptive algorithm
Online AccessGet full text

Cover

More Information
Summary:We study a linear stochastic approximation algorithm that arises in the context of reinforcement learning. The algorithm employs a decreasing step-size, and is driven by Markov noise with time-varying statistics. We show that under suitable conditions, the algorithm can track the changes in the statistics of the Markov noise, as long as these changes are slower than the rate at which the step-size of the algorithm goes to zero.
ISSN:0167-6911
1872-7956
DOI:10.1016/S0167-6911(03)00132-4