From differential equation solvers to accelerated first-order methods for convex optimization

• Published in: Mathematical programming Vol. 195; no. 1-2; pp. 735 - 781
• Main Authors: Luo, Hao, Chen, Long
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.09.2022; Springer; Springer Nature B.V
• Subjects: Algorithms; Calculus of Variations and Optimal Control; Optimization; Combinatorics; Computational geometry; Convergence; Convexity; Differential equations; Flow stability; Full Length Paper; Liapunov functions; Mathematical and Computational Physics; Mathematical Methods in Physics; Mathematics; Mathematics and Statistics; Mathematics of Computing; Methods; Numerical Analysis; Optimization; Solvers; Theoretical; Accelerated first-order methods; 65B99; Convex optimization; 90C25; 65L20; Ordinary differential equation; Exponential decay; 37N40; Convergence analysis; Lyapunov function
• ISSN: 0025-5610; 1436-4646
• DOI: 10.1007/s10107-021-01713-3

Convergence analysis of accelerated first-order methods for convex optimization problems are developed from the point of view of ordinary differential equation solvers. A new dynamical system, called Nesterov accelerated gradient (NAG) flow, is derived from the connection between acceleration mechanism and A -stability of ODE solvers, and the exponential decay of a tailored Lyapunov function along with the solution trajectory is proved. Numerical discretizations of NAG flow are then considered and convergence rates are established via a discrete Lyapunov function. The proposed differential equation solver approach can not only cover existing accelerated methods, such as FISTA, Güler’s proximal algorithm and Nesterov’s accelerated gradient method, but also produce new algorithms for composite convex optimization that possess accelerated convergence rates. Both the convex and the strongly convex cases are handled in a unified way in our approach.