Offline RL via Feature-Occupancy Gradient Ascent

We study offline Reinforcement Learning in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linear-program formulation of the optimal control problem in MDPs, we devel...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Neu, Gergely, Okolo, Nneka
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 22.05.2024
Subjects
Algorithms
Complexity
Covariance matrix
Feature maps
Markov processes
Optimal control
Optimization
Upper bounds
Online AccessGet full text

Cover

More Information
Summary:We study offline Reinforcement Learning in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linear-program formulation of the optimal control problem in MDPs, we develop a new algorithm that performs a form of gradient ascent in the space of feature occupancies, defined as the expected feature vectors that can potentially be generated by executing policies in the environment. We show that the resulting simple algorithm satisfies strong computational and sample complexity guarantees, achieved under the least restrictive data coverage assumptions known in the literature. In particular, we show that the sample complexity of our method scales optimally with the desired accuracy level and depends on a weak notion of coverage that only requires the empirical feature covariance matrix to cover a single direction in the feature space (as opposed to covering a full subspace). Additionally, our method is easy to implement and requires no prior knowledge of the coverage ratio (or even an upper bound on it), which altogether make it the strongest known algorithm for this setting to date.
ISSN:2331-8422