Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
Specifying reward functions for complex tasks like object manipulation or driving is challenging to do by hand. Reward learning seeks to address this by learning a reward model using human feedback on selected query policies. This shifts the burden of reward specification to the optimal design of th...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
23.02.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Specifying reward functions for complex tasks like object manipulation or
driving is challenging to do by hand. Reward learning seeks to address this by
learning a reward model using human feedback on selected query policies. This
shifts the burden of reward specification to the optimal design of the queries.
We propose a theoretical framework for studying reward learning and the
associated optimal experiment design problem. Our framework models rewards and
policies as nonparametric functions belonging to subsets of Reproducing Kernel
Hilbert Spaces (RKHSs). The learner receives (noisy) oracle access to a true
reward and must output a policy that performs well under the true reward. For
this setting, we first derive non-asymptotic excess risk bounds for a simple
plug-in estimator based on ridge regression. We then solve the query design
problem by optimizing these risk bounds with respect to the choice of query set
and obtain a finite sample statistical rate, which depends primarily on the
eigenvalue spectrum of a certain linear operator on the RKHSs. Despite the
generality of these results, our bounds are stronger than previous bounds
developed for more specialized problems. We specifically show that the
well-studied problem of Gaussian process (GP) bandit optimization is a special
case of our framework, and that our bounds either improve or are competitive
with known regret guarantees for the Mat\'ern kernel. |
---|---|
DOI: | 10.48550/arxiv.2302.12349 |