On Minimizing Total Discounted Cost in MDPs Subject to Reachability Constraints

• Published in: IEEE transactions on automatic control Vol. 69; no. 9; pp. 6466 - 6473
• Main Authors: Savas, Yagiz, Verginis, Christos K., Hibbard, Michael, Topcu, Ufuk
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Approximation algorithms; Costs; Criteria; Discounting; Integer programming; Linear programming; Markov decision processes; Markov decision processes (MDPs); Markov processes; Mixed integer; optimization; Planning; Probabilistic logic; reachability; Reagents; Synthesis; Task analysis; Trajectory; Transient performance
• ISSN: 0018-9286; 1558-2523
• DOI: 10.1109/TAC.2024.3384834

In this article, we study the synthesisof a policy in a Markov decision process (MDP) following which an agent reaches a target state in the MDP while minimizing its total discounted cost. The problem combines a reachability criterion with a discounted cost criterion and naturally expresses the completion of a task with probabilistic guarantees and optimal transient performance. We first establish that an optimal policy for the considered formulation may not exist but that there always exists a near-optimal stationary policy. We additionally provide a necessary and sufficient condition for the existence of an optimal policy. We then restrict our attention to stationary deterministic policies and show that the decision problem associated with the synthesis of an optimal stationary deterministic policy is NP-complete. Finally, we provide an exact algorithm based on mixed-integer linear programming and propose an efficient approximation algorithm based on linear programming for the synthesis of an optimal stationary deterministic policy.