Minimization of a class of rare event probabilities and buffered probabilities of exceedance

• Published in: Annals of operations research Vol. 302; no. 1; pp. 49 - 83
• Main Authors: Budhiraja, Amarjit, Lu, Shu, Yu, Yang, Tran-Dinh, Quoc
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.07.2021; Springer; Springer Nature B.V
• Subjects: Algorithms; Analysis; Asymptotic properties; Buffers; Business and Management; Combinatorial probabilities; Combinatorics; Computational geometry; Convexity; Design parameters; Deviation; Estimation; Geometric probabilities; Importance sampling; Mathematical optimization; Monte Carlo simulation; Operations research; Operations Research/Decision Theory; Optimization; Original Research; Probabilities; Probability; Random variables; Theory of Computation; United States; 60F10; 65K10; 90C15; Large deviations; Buffered probability; 65C05; Stochastic optimization; Importance sampling
• ISSN: 0254-5330; 1572-9338
• DOI: 10.1007/s10479-021-03991-8

We consider the problem of choosing design parameters to minimize the probability of an undesired rare event that is described through the average of n i.i.d. random variables. Since the probability of interest for near optimal design parameters is very small, one needs to develop suitable accelerated Monte-Carlo methods for estimating its value. One of the challenges in the study is that simulating from exponential twists of the laws of the summands may be computationally demanding since these transformed laws may be non-standard and intractable. We consider a setting where the summands are given as a nonlinear functional of random variables, the exponential twists of whose distributions take a simpler form than those for the original summands. We use techniques from Dupuis and Wang (Stochastics 76(6):481–508, 2004, Math Oper Res 32(3):723–757, 2007) to identify the appropriate Issacs equations whose subsolutions are used to construct tractable importance sampling (IS) schemes. We also study the closely related problem of estimating buffered probability of exceedance and provide the first rigorous results that relate the asymptotics of buffered probability and that of the ordinary probability under a large deviation scaling. The analogous minimization problem for buffered probability, under conditions, can be formulated as a convex optimization problem. We show that, under conditions, changes of measures that are asymptotically efficient (under the large deviation scaling) for estimating ordinary probability are also asymptotically efficient for estimating the buffered probability of exceedance. We embed the constructed IS scheme in gradient descent algorithms to solve the optimization problems, and illustrate these schemes through computational experiments.