What is decidable about partially observable Markov decision processes with ω-regular objectives

• Published in: Journal of computer and system sciences Vol. 82; no. 5; pp. 878 - 911
• Main Authors: Chatterjee, Krishnendu, Chmelík, Martin, Tracol, Mathieu
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.08.2016
• Subjects: Complexity; Finite-memory strategies; Languages; Markov decision processes; Markov processes; Optimization; Parity; Parity objectives; Partially observable Markov decision processes (POMDPs); Qualitative analysis; Specifications; Strategy; ω-Regular conditions; Parity objectives; Finite-memory strategies; ω-Regular conditions; Markov decision processes; Partially observable Markov decision processes (POMDPs)
• ISSN: 0022-0000; 1090-2724
• DOI: 10.1016/j.jcss.2016.02.009

•Decidability of qualitative analysis of parity POMDPs under finite-memory strategies.•Optimal memory bounds and complexity (EXPTIME-completeness) for the above problem.•Implementation of our algorithm with several heuristics and experimental results. We consider partially observable Markov decision processes (POMDPs) with ω-regular conditions specified as parity objectives. The class of ω-regular languages provides a robust specification language to express properties in verification, and parity objectives are canonical forms to express them. The qualitative analysis problem given a POMDP and a parity objective asks whether there is a strategy to ensure that the objective is satisfied with probability 1 (resp. positive probability). While the qualitative analysis problems are undecidable even for special cases of parity objectives, we establish decidability (with optimal complexity) for POMDPs with all parity objectives under finite-memory strategies. We establish optimal (exponential) memory bounds and EXPTIME-completeness of the qualitative analysis problems under finite-memory strategies for POMDPs with parity objectives. We also present a practical approach, where we design heuristics to deal with the exponential complexity, and have applied our implementation on a number of POMDP examples.