Near-Optimal Regret Bounds for Thompson Sampling

• Published in: Journal of the ACM Vol. 64; no. 5; pp. 1 - 24
• Main Authors: Agrawal, Shipra, Goyal, Navin
• Format: Journal Article
• Language: English
• Published: New York Association for Computing Machinery 01.10.2017
• Subjects: Bayesian analysis; Empirical analysis; Heuristic; Lower bounds; Martingales; Optimization; Probability distribution functions; Randomized algorithms; Sampling; Studies
• ISSN: 0004-5411; 1557-735X
• DOI: 10.1145/3088510

Thompson Sampling (TS) is one of the oldest heuristics for multiarmed bandit problems. It is a randomized algorithm based on Bayesian ideas and has recently generated significant interest after several studies demonstrated that it has favorable empirical performance compared to the state-of-the-art methods. In this article, a novel and almost tight martingale-based regret analysis for Thompson Sampling is presented. Our technique simultaneously yields both problem-dependent and problem-independent bounds: (1) the first near-optimal problem-independent bound of O (√ NT ln T ) on the expected regret and (2) the optimal problem-dependent bound of (1 + ϵ)Σ i ln T / d (μ i ,μ 1 ) + O ( N /ϵ 2 ) on the expected regret (this bound was first proven by Kaufmann et al. (2012b)). Our technique is conceptually simple and easily extends to distributions other than the Beta distribution used in the original TS algorithm. For the version of TS that uses Gaussian priors, we prove a problem-independent bound of O (√ NT ln N ) on the expected regret and show the optimality of this bound by providing a matching lower bound. This is the first lower bound on the performance of a natural version of Thompson Sampling that is away from the general lower bound of Ω (√ NT ) for the multiarmed bandit problem.