FedProc: Prototypical contrastive federated learning on non-IID data

• Published in: Future generation computer systems Vol. 143; pp. 93 - 104
• Main Authors: Mu, Xutong, Shen, Yulong, Cheng, Ke, Geng, Xueli, Fu, Jiaxuan, Zhang, Tao, Zhang, Zhiwei
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.06.2023
• Subjects: Contrastive learning; Federated learning; Non-IID data; Prototypes; Prototypical contrastive loss; Prototypical networks; Federated learning; Prototypes; Prototypical contrastive loss; Non-IID data; Contrastive learning; Prototypical networks
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2023.01.019

Federated learning (FL) enables multiple clients to jointly train high-performance deep learning models while maintaining the training data locally. However, it is challenging to accomplish this form of efficient collaborative learning when all the clients’ local data are not independent and identically distributed (i.e., non-IID). Despite extensive efforts to address this challenge, the results for image classification tasks remain inadequate. In this paper, we propose FedProc: prototypical contrastive federated learning. The core idea of this approach is to utilize the prototypes as global knowledge to correct the drift of each client’s local training. Specifically, we designed a local network structure and a global prototype contrast loss to regulate the training of the local model. These efforts make the direction of local optimization consistent with the global optimum such that the global model achieves good performance on non-IID data. Evaluative studies supported by theoretical significance demonstrate that FedProc improves accuracy by 1.6% to 7.9% with an acceptable computational cost compared to state-of-the-art federated learning methods. •This paper proposes a novel federated learning framework (FedProc) to solve the non-IID data problem. The framework innovatively introduces a global class prototype to correct for local training, making the direction of local optimization consistent with the global optimization goal.•In this paper, we elaborate a generic hybrid local network architecture such that the local network takes full advantage of the potential knowledge provided by the global class prototype. The superior performance of this architecture is due to the design of a hybrid loss function.•We theoretically analyze the convergence of FedProc and obtain an upper bound on the convergence, which provides convergence guarantees for this work. In addition, experimental results show that FedProc is significantly better than the state-of-the-art methods in terms of accuracy and computational efficiency.