Lower bounds for finding stationary points I

• Published in: Mathematical programming Vol. 184; no. 1-2; pp. 71 - 120
• Main Authors: Carmon, Yair, Duchi, John C., Hinder, Oliver, Sidford, Aaron
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.11.2020; Springer Nature B.V
• Subjects: Algorithms; Calculus of Variations and Optimal Control; Optimization; Combinatorics; Complexity; Convex analysis; Derivatives; Electrical engineering; Full Length Paper; Lower bounds; Mathematical and Computational Physics; Mathematical Methods in Physics; Mathematical programming; Mathematics; Mathematics and Statistics; Mathematics of Computing; Numerical Analysis; Optimization; Queries; Regularization; Regularization methods; Theoretical; Dimension-free rates; Gradient descent; Cubic regularization of Newton’s method; 68Q25; Non-convex optimization; 90C30; Information-based complexity; 90C06; 90C26; 90C60
• ISSN: 0025-5610; 1436-4646
• DOI: 10.1007/s10107-019-01406-y

We prove lower bounds on the complexity of finding ϵ -stationary points (points x such that ‖ ∇ f ( x ) ‖ ≤ ϵ ) of smooth, high-dimensional, and potentially non-convex functions f . We consider oracle-based complexity measures, where an algorithm is given access to the value and all derivatives of f at a query point x . We show that for any (potentially randomized) algorithm A , there exists a function f with Lipschitz p th order derivatives such that A requires at least ϵ - ( p + 1 ) / p queries to find an ϵ -stationary point. Our lower bounds are sharp to within constants, and they show that gradient descent, cubic-regularized Newton’s method, and generalized p th order regularization are worst-case optimal within their natural function classes.