Lower complexity bounds of first-order methods for convex-concave bilinear saddle-point problems

• Published in: Mathematical programming Vol. 185; no. 1-2; pp. 1 - 35
• Main Authors: Ouyang, Yuyuan, Xu, Yangyang
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.01.2021; Springer Nature B.V
• Subjects: Calculus of Variations and Optimal Control; Optimization; Combinatorics; Complexity; Convergence; Convex analysis; Convexity; Full Length Paper; Mathematical and Computational Physics; Mathematical Methods in Physics; Mathematical programming; Mathematics; Mathematics and Statistics; Mathematics of Computing; Methods; Numerical Analysis; Optimization; Production methods; Saddle points; Theoretical; 68Q25; Information-based complexity; Convex optimization; 90C25; 90C06; Lower complexity bound; 49M37; 90C60; First-order methods; Saddle point problems
• ISSN: 0025-5610; 1436-4646
• DOI: 10.1007/s10107-019-01420-0

On solving a convex-concave bilinear saddle-point problem (SPP), there have been many works studying the complexity results of first-order methods. These results are all about upper complexity bounds, which can determine at most how many iterations would guarantee a solution of desired accuracy. In this paper, we pursue the opposite direction by deriving lower complexity bounds of first-order methods on large-scale SPPs. Our results apply to the methods whose iterates are in the linear span of past first-order information, as well as more general methods that produce their iterates in an arbitrary manner based on first-order information. We first work on the affinely constrained smooth convex optimization that is a special case of SPP. Different from gradient method on unconstrained problems, we show that first-order methods on affinely constrained problems generally cannot be accelerated from the known convergence rate O (1 /  t ) to O ( 1 / t 2 ) , and in addition, O (1 /  t ) is optimal for convex problems. Moreover, we prove that for strongly convex problems, O ( 1 / t 2 ) is the best possible convergence rate, while it is known that gradient methods can have linear convergence on unconstrained problems. Then we extend these results to general SPPs. It turns out that our lower complexity bounds match with several established upper complexity bounds in the literature, and thus they are tight and indicate the optimality of several existing first-order methods.