Exponential asymptotic optimality of Whittle index policy

• Published in: Queueing systems Vol. 104; no. 1-2; pp. 107 - 150
• Main Authors: Gast, Nicolas, Gaujal, Bruno, Yan, Chen
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.06.2023; Springer Nature B.V; Springer Verlag
• Subjects: Approximation; Asymptotic properties; Business and Management; Computer Communication Networks; Computer Science; Continuous time systems; Control; Convergence; Dynamical systems; Fixed points (mathematics); Mathematics; Neighborhoods; Operations Research/Decision Theory; Optimization; Optimization and Control; Ordinary differential equations; Performance evaluation; Probability; Probability Theory and Stochastic Processes; Stochastic systems; Supply Chain Management; Systems Theory; Multi-armed bandits; Whittle index; Asymptotic optimality; 90B18; 90C40; 90C05; Whittle Index; secondary 37H12; 60F10; MSC2020 subject classifications: Primary 90C40; 68M20 Multi-armed Bandits; Asymptotic Optimality
• ISSN: 0257-0130; 1572-9443
• DOI: 10.1007/s11134-023-09875-x

We evaluate the performance of Whittle index policy for restless Markovian bandit. It is shown in Weber and Weiss (J Appl Probab 27(3):637–648, 1990) that if the bandit is indexable and the associated deterministic system has a global attractor fixed point, then the Whittle index policy is asymptotically optimal in the regime where the arm population grows proportionally with the number of activation arms. In this paper, we show that, under the same conditions, this convergence rate is exponential in the arm population, unless the fixed point is singular (to be defined later), which almost never happens in practice. Our result holds for the continuous-time model of Weber and Weiss (1990) and for a discrete-time model in which all bandits make synchronous transitions. Our proof is based on the nature of the deterministic equation governing the stochastic system: We show that it is a piecewise affine continuous dynamical system inside the simplex of the empirical measure of the arms. Using simulations and numerical solvers, we also investigate the singular cases, as well as how the level of singularity influences the (exponential) convergence rate. We illustrate our theorem on a Markovian fading channel model.