Benchmark for Personalized Federated Learning

• Published in: IEEE open journal of the Computer Society Vol. 5; pp. 1 - 12
• Main Authors: Matsuda, Koji, Sasaki, Yuya, Xiao, Chuan, Onizuka, Makoto
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Benchmark testing; Benchmarking; Benchmarks; Clients; Computational modeling; Computer vision; Customization; Data models; Datasets; Distributed Computing; Federated Learning; Heterogeneity; Machine learning; Natural language processing; Servers; Task analysis; Training
• DOI: 10.1109/OJCS.2023.3332351
• ISSN: 2644-1268

Federated learning is a distributed machine learning approach that allows a single server to collaboratively build machine learning models with multiple clients without sharing datasets. Since data distributions may differ across clients, data heterogeneity is a challenging issue in federated learning. To address this issue, numerous federated learning methods have been proposed to build personalized models for clients, referred to as personalized federated learning. Nevertheless, no studies comprehensively investigate the performance of personalized federated learning methods in various experimental settings such as datasets and client settings. Therefore, in this article, we aim to benchmark the performance of existing personalized federated learning methods in various settings. We first survey the experimental settings in existing studies. We then benchmark the performance of existing methods through comprehensive experiments to reveal their characteristics in computer vision and natural language processing tasks which are the most popular tasks based on our survey. Our experimental study shows that (i) large data heterogeneity often leads to highly accurate predictions and (ii) standard federated learning methods (e.g. FedAvg) with fine-tuning often outperform personalized federated learning methods.