A Flexible ADMM Algorithm for Big Data Applications

• Published in: Journal of scientific computing Vol. 71; no. 1; pp. 435 - 467
• Main Authors: Robinson, Daniel P., Tappenden, Rachael
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2017; Springer Nature B.V
• Subjects: Algorithms; Big Data; Computational Mathematics and Numerical Analysis; Convergence; Convex analysis; Lagrange multiplier; Linear programming; Mathematical and Computational Engineering; Mathematical and Computational Physics; Mathematics; Mathematics and Statistics; Methods; Optimization; Regularization; Theoretical; Variables; 58C15; 68Q25; 65K05; Jacobi; 65Y20; Separable function; Gauss–Seidel; 90C30; Convex optimization; 65K10; 49M15; 49M37; 90C60; ADMM; Regularization
• ISSN: 0885-7474; 1573-7691
• DOI: 10.1007/s10915-016-0306-6

We present a Flexible Alternating Direction Method of Multipliers (F-ADMM) algorithm for solving optimization problems involving a strongly convex objective function that is separable into n ≥ 2 blocks, subject to (non-separable) linear equality constraints. The F-ADMM algorithm uses a Gauss–Seidel scheme to update blocks of variables, and a regularization term is added to each of the subproblems. We prove, under common assumptions, that F-ADMM is globally convergent and that the iterates converge linearly. We also present a special case of F-ADMM that is partially parallelizable , which makes it attractive in a big data setting. In particular, we partition the data into groups, so that each group consists of multiple blocks of variables. By applying F-ADMM to this partitioning of the data, and using a specific regularization matrix, we obtain a hybrid ADMM (H-ADMM) algorithm: the grouped data is updated in a Gauss–Seidel fashion, and the blocks within each group are updated in a Jacobi (parallel) manner. Convergence of H-ADMM follows directly from the convergence properties of F-ADMM. Also, we describe a special case of H-ADMM that may be applied to functions that are convex, rather than strongly convex. Numerical experiments demonstrate the practical advantages of these new algorithms.