RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk

Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Jia Lin Hau, Petrik, Marek, Ghavamzadeh, Mohammad, Reazul Russel
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 14.09.2022
Subjects
Algorithms
Empirical analysis
Horizon
Risk levels
Robustness
Time dependence
Transition probabilities
Uncertainty
Online AccessGet full text

Cover

More Information
Summary:Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs. We call this framework that combines Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion is defined using either EVaR or the entropic risk, the optimal policy in RASR can be computed efficiently using a new dynamic program formulation with a time-dependent risk level. As a result, the optimal risk-averse policies are deterministic but time-dependent, even in the infinite-horizon discounted setting. We also show that particular RASR objectives reduce to risk-averse RL with mean posterior transition probabilities. Our empirical results show that our new algorithms consistently mitigate uncertainty as measured by EVaR and other standard risk measures.
ISSN:2331-8422