Constrained regret minimization for multi-criterion multi-armed bandits

• Published in: Machine learning Vol. 112; no. 2; pp. 431 - 458
• Main Authors: Kagrecha, Anmol, Nair, Jayakrishnan, Jagannathan, Krishna
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2023; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Boolean; Computer Science; Constraints; Control; Decision making; Distance learning; Energy consumption; Feasibility; Logarithms; Machine Learning; Mechatronics; Multi-armed bandit problems; Natural Language Processing (NLP); Optimization; Performance evaluation; Robotics; Simulation and Modeling; Special Issue of the ECML PKDD 2022 Journal Track; Multi-criterion multi-armed bandits; Constrained bandits; Regret minimization
• ISSN: 0885-6125; 1573-0565
• DOI: 10.1007/s10994-022-06291-9

We consider a stochastic multi-armed bandit setting and study the problem of constrained regret minimization over a given time horizon. Each arm is associated with an unknown, possibly multi-dimensional distribution, and the merit of an arm is determined by several, possibly conflicting attributes. The aim is to optimize a ‘primary’ attribute subject to user-provided constraints on other ‘secondary’ attributes. We assume that the attributes can be estimated using samples from the arms’ distributions, and that the estimators enjoy suitable concentration properties. We propose an algorithm called Con-LCB that guarantees a logarithmic regret, i.e., the average number of plays of all non-optimal arms is at most logarithmic in the horizon. The algorithm also outputs a boolean flag that correctly identifies, with high probability, whether the given instance is feasible/infeasible with respect to the constraints. We also show that Con-LCB is optimal within a universal constant, i.e., that more sophisticated algorithms cannot do much better universally. Finally, we establish a fundamental trade-off between regret minimization and feasibility identification. Our framework finds natural applications, for instance, in financial portfolio optimization, where risk constrained maximization of expected return is meaningful.