Lower bounds for finding stationary points I - Carmon, Yair; Duchi, John C.; Hinder, Oliver; Sidford, Aaron (Mathematical programming, 2020)
We prove lower bounds on the complexity of finding ϵ -stationary points (points x such that ‖ ∇ f ( x ) ‖ ≤ ϵ ) of smooth, high-dimensional, and potentially non-convex functions f . We consider oracle-based complexity measures, where an algorithm is given access to the value and all derivatives of f at a query point x . We show that for any (potentially randomized) algorithm A , there exists a function f with Lipschitz p th order derivatives such that A requires at least ϵ - ( p + 1 ) / p queries to find an ϵ -stationary point. Our lower bounds are sharp to within constants, and they show that gradient descent, cubic-regularized Newton’s method, and generalized p th order regularization are worst-case optimal within their natural function classes.