Optimizing the Efficiency of First-Order Methods for Decreasing the Gradient of Smooth Convex Functions

• Published in: Journal of optimization theory and applications Vol. 188; no. 1; pp. 192 - 219
• Main Authors: Kim, Donghwan, Fessler, Jeffrey A.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.01.2021; Springer Nature B.V
• Subjects: Applications of Mathematics; Calculus of Variations and Optimal Control; Optimization; Cost function; Engineering; Mathematics; Mathematics and Statistics; Operations Research/Decision Theory; Optimization; Theory of Computation; Gradient methods; 68Q25; 90C30; 90C25; 90C22; Smooth convex minimization; 90C60; 49M25; First-order methods; Worst-case performance analysis
• ISSN: 0022-3239; 1573-2878
• DOI: 10.1007/s10957-020-01770-2

This paper optimizes the step coefficients of first-order methods for smooth convex minimization in terms of the worst-case convergence bound ( i.e. , efficiency) of the decrease in the gradient norm. This work is based on the performance estimation problem approach. The worst-case gradient bound of the resulting method is optimal up to a constant for large-dimensional smooth convex minimization problems, under the initial bounded condition on the cost function value. This paper then illustrates that the proposed method has a computationally efficient form that is similar to the optimized gradient method.