Constant step stochastic approximations involving differential inclusions: stability, long-run convergence and applications

• Published in: Stochastics (Abingdon, Eng. : 2005) Vol. 91; no. 2; pp. 288 - 320
• Main Authors: Bianchi, Pascal, Hachem, Walid, Salim, Adil
• Format: Journal Article
• Language: English
• Published: Abingdon Taylor & Francis 17.02.2019; Taylor & Francis Ltd; Taylor & Francis: STM, Behavioural Science and Public Health Titles
• Subjects: Convergence; Differential inclusions; dynamical systems; Markov analysis; Markov chains; Mathematics; non-convex optimization; Probability; queueing systems; Queues; Stability; stochastic approximation with constant step; Non-convex optimization; Differential inclusions; Stochastic approximation with constant step; 47N10; 54C60; Dynamical systems; 62L20; Queueing systems AMS Subject Classification: 34A60
• ISSN: 1744-2508; 1744-2516
• DOI: 10.1080/17442508.2018.1539086

We consider a Markov chain whose kernel is indexed by a scaling parameter , referred to as the step size. The aim is to analyze the behaviour of the Markov chain in the doubly asymptotic regime where then . First, under mild assumptions on the so-called drift of the Markov chain, we show that the interpolated process converges narrowly to the solutions of a Differential Inclusion (DI) involving an upper semicontinuous set-valued map with closed and convex values. Second, we provide verifiable conditions which ensure the stability of the iterates. Third, by putting the above results together, we establish the long run convergence of the iterates as , to the Birkhoff center of the DI. The ergodic behaviour of the iterates is also provided. Application examples are investigated. We apply our findings to (1) the problem of nonconvex proximal stochastic optimization and (2) a fluid model of parallel queues.