Parallel Multi-Block ADMM with o(1 / k) Convergence

• Published in: Journal of scientific computing Vol. 71; no. 2; pp. 712 - 736
• Main Authors: Deng, Wei, Lai, Ming-Jun, Peng, Zhimin, Yin, Wotao
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2017; Springer Nature B.V
• Subjects: Algorithms; Computational Mathematics and Numerical Analysis; Convergence; Convex analysis; Decomposition; Guarantees; Lagrange multiplier; Mathematical analysis; Mathematical and Computational Engineering; Mathematical and Computational Physics; Mathematics; Mathematics and Statistics; Matrices (mathematics); Optimization; Parameters; Theoretical; Variables; Convergence rate; Alternating direction method of multipliers; Parallel and distributed computing; ADMM
• ISSN: 0885-7474; 1573-7691
• DOI: 10.1007/s10915-016-0318-2

This paper introduces a parallel and distributed algorithm for solving the following minimization problem with linear constraints: minimize f 1 ( x 1 ) + ⋯ + f N ( x N ) subject to A 1 x 1 + ⋯ + A N x N = c , x 1 ∈ X 1 , … , x N ∈ X N , where N ≥ 2 , f i are convex functions, A i are matrices, and X i are feasible sets for variable x i . Our algorithm extends the alternating direction method of multipliers (ADMM) and decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This paper shows that the classic ADMM can be extended to the N -block Jacobi fashion and preserve convergence in the following two cases: (i) matrices A i are mutually near-orthogonal and have full column-rank, or (ii) proximal terms are added to the N subproblems (but without any assumption on matrices A i ). In the latter case, certain proximal terms can let the subproblem be solved in more flexible and efficient ways. We show that ‖ x k + 1 - x k ‖ M 2 converges at a rate of o (1 /  k ) where M is a symmetric positive semi-definte matrix. Since the parameters used in the convergence analysis are conservative, we introduce a strategy for automatically tuning the parameters to substantially accelerate our algorithm in practice. We implemented our algorithm (for the case ii above) on Amazon EC2 and tested it on basis pursuit problems with >300 GB of distributed data. This is the first time that successfully solving a compressive sensing problem of such a large scale is reported.