Communication-Efficient Federated Learning via Predictive Coding

• Published in: IEEE journal of selected topics in signal processing Vol. 16; no. 3; pp. 369 - 380
• Main Authors: Yue, Kai, Jin, Richeng, Wong, Chau-Wai, Dai, Huaiyu
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.04.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Coding; Collaborative work; Communication; Costs; Data compression; distributed optimization; Electronic devices; Entropy coding; Federated learning; Machine learning; Predictive coding; Predictive models; Quantization (signal); Redundancy; Servers; Wireless communications
• ISSN: 1932-4553; 1941-0484
• DOI: 10.1109/JSTSP.2022.3142678

Federated learning can enable remote workers to collaboratively train a shared machine learning model while allowing training data to be kept locally. In the use case of wireless mobile devices, the communication overhead is a critical bottleneck due to limited power and bandwidth. Prior work has utilized various data compression tools such as quantization and sparsification to reduce the overhead. In this paper, we propose a predictive coding based compression scheme for federated learning. The scheme has shared prediction functions among all devices and allows each worker to transmit a compressed residual vector derived from the reference. In each communication round, we select the predictor and quantizer based on the rate-distortion cost, and further reduce the redundancy with entropy coding. Extensive simulations reveal that the communication cost can be reduced up to 99% with even better learning performance when compared with other baseline methods.