Efficiency of minimizing compositions of convex functions and smooth maps

• Published in: Mathematical programming Vol. 178; no. 1-2; pp. 503 - 558
• Main Authors: Drusvyatskiy, D., Paquette, C.
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.11.2019; Springer Nature B.V
• Subjects: Algorithms; Calculus of Variations and Optimal Control; Optimization; Combinatorics; Composition; Convexity; Efficiency; Full Length Paper; Mathematical and Computational Physics; Mathematical Methods in Physics; Mathematics; Mathematics and Statistics; Mathematics of Computing; Numerical Analysis; Theoretical; Secondary 90C06; Fast gradient methods; Inexactness; Gauss–Newton; Complexity; Prox-gradient; Incremental methods; Primary 97N60; 90C30; Composite minimization; Smoothing; 90C25; Acceleration
• DOI: 10.1007/s10107-018-1311-3
• ISSN: 0025-5610

We consider global efficiency of algorithms for minimizing a sum of a convex function and a composition of a Lipschitz convex function with a smooth map. The basic algorithm we rely on is the prox-linear method, which in each iteration solves a regularized subproblem formed by linearizing the smooth map. When the subproblems are solved exactly, the method has efficiency O ( ε - 2 ) , akin to gradient descent for smooth minimization. We show that when the subproblems can only be solved by first-order methods, a simple combination of smoothing, the prox-linear method, and a fast-gradient scheme yields an algorithm with complexity O ~ ( ε - 3 ) . We round off the paper with an inertial prox-linear method that automatically accelerates in presence of convexity.