Accelerating Federated Learning via Momentum Gradient Descent

• Published in: IEEE transactions on parallel and distributed systems Vol. 31; no. 8; pp. 1754 - 1766
• Main Authors: Liu, Wei, Chen, Li, Chen, Yunfei, Zhang, Wenyi
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accelerating convergence; Acceleration; Computational modeling; Computer simulation; Convergence; Data models; Distributed databases; distributed machine learning; Federated learning; Iterative methods; Machine learning; Momentum; momentum gradient descent; Servers; Upper bounds
• ISSN: 1045-9219; 1558-2183
• DOI: 10.1109/TPDS.2020.2975189

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update which can potentially accelerate convergence. In this article, we consider momentum term which relates to the last iteration. The proposed momentum federated learning (MFL) uses momentum gradient descent (MGD) in the local update step of FL system. We establish global convergence properties of MFL and derive an upper bound on MFL convergence rate. Comparing the upper bounds on MFL and FL convergence rates, we provide conditions in which MFL accelerates the convergence. For different machine learning models, the convergence performance of MFL is evaluated based on experiments with MNIST and CIFAR-10 datasets. Simulation results confirm that MFL is globally convergent and further reveal significant convergence improvement over FL.