Interactive Multi-objective Reinforcement Learning in Multi-armed Bandits with Gaussian Process Utility Models

• Published in: Machine Learning and Knowledge Discovery in Databases Vol. 12459; pp. 463 - 478
• Main Authors: Roijers, Diederik M., Zintgraf, Luisa M., Libin, Pieter, Reymond, Mathieu, Bargiacchi, Eugenio, Nowé, Ann
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Multi-armed bandits; Multiple objectives; Reinforcement learning; Thompson sampling
• ISBN: 3030676633; 9783030676636
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-67664-3_28

In interactive multi-objective reinforcement learning (MORL), an agent has to simultaneously learn about the environment and the preferences of the user, in order to quickly zoom in on those decisions that are likely to be preferred by the user. In this paper we study interactive MORL in the context of multi-objective multi-armed bandits. Contrary to earlier approaches to interactive MORL that force the utility of the user to be expressed as a weighted sum of the values for each objective, we do not make such stringent a priori assumptions. Specifically, we not only allow non-linear preferences, but also obviate the need to specify the exact model class in the utility function must fall. To achieve this, we propose a new approach called Gaussian-process Utility Thompson Sampling (GUTS). GUTS employs parameterless Bayesian learning to allow any type of utility function, exploits monotonicity information, and limits the number of queries posed to the user by ensuring that questions are statistically significant. We show empirically that GUTS can learn non-linear preferences, and that the regret and number of queries posed to the user are highly sub-linear in the number of arm pulls. (A preliminary version of this work was presented at the ALA workshop in 2018 [20]).