A Bayesian Approach to Online Planning
The combination of Monte Carlo tree search and neural networks has revolutionized online planning. As neural network approximations are often imperfect, we ask whether uncertainty estimates about the network outputs could be used to improve planning. We develop a Bayesian planning approach that faci...
Saved in:
Main Authors | , , , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
04.06.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | The combination of Monte Carlo tree search and neural networks has
revolutionized online planning. As neural network approximations are often
imperfect, we ask whether uncertainty estimates about the network outputs could
be used to improve planning. We develop a Bayesian planning approach that
facilitates such uncertainty quantification, inspired by classical ideas from
the meta-reasoning literature. We propose a Thompson sampling based algorithm
for searching the tree of possible actions, for which we prove the first (to
our knowledge) finite time Bayesian regret bound, and propose an efficient
implementation for a restricted family of posterior distributions. In addition
we propose a variant of the Bayes-UCB method applied to trees. Empirically, we
demonstrate that on the ProcGen Maze and Leaper environments, when the
uncertainty estimates are accurate but the neural network output is inaccurate,
our Bayesian approach searches the tree much more effectively. In addition, we
investigate whether popular uncertainty estimation methods are accurate enough
to yield significant gains in planning. Our code is available at:
https://github.com/nirgreshler/bayesian-online-planning. |
---|---|
DOI: | 10.48550/arxiv.2406.02103 |