Collaborative deep learning across multiple data centers

Valuable training data is often owned by independent organizations and located in multiple data centers. Most deep learning approaches require to centralize the multi-datacenter data for performance purpose. In practice, however, it is often infeasible to transfer all data of different organizations...

Full description

Saved in:
Bibliographic Details
Published inScience China. Information sciences Vol. 63; no. 8; p. 182102
Main Authors Mi, Haibo, Xu, Kele, Feng, Dawei, Wang, Huaimin, Zhang, Yiming, Zheng, Zibin, Chen, Chuan, Lan, Xu
Format Journal Article
LanguageEnglish
Published Beijing Science China Press 01.08.2020
Springer Nature B.V
Subjects
Artificial neural networks
Business competition
Computer centers
Computer Science
Data centers
Deep learning
Information Systems and Communication Service
Machine learning
Organizations
Privacy
Research Paper
multiple datacenters
distributed machine learning
collaborative learning
Online AccessGet full text

Cover

More Information
Summary:Valuable training data is often owned by independent organizations and located in multiple data centers. Most deep learning approaches require to centralize the multi-datacenter data for performance purpose. In practice, however, it is often infeasible to transfer all data of different organizations to a centralized data center owing to the constraints of privacy regulations. It is very challenging to conduct the geo-distributed deep learning among data centers without the privacy leaks. Model averaging is a conventional choice for data parallelized training and can reduce the risk of privacy leaks, but its ineffectiveness is claimed by previous studies as deep neural networks are often non-convex. In this paper, we argue that model averaging can be effective in the decentralized environment by using two strategies, namely, the cyclical learning rate (CLR) and the increased number of epochs for local model training. With the two strategies, we show that model averaging can provide competitive performance in the decentralized mode compared to the data-centralized one. In a practical environment with multiple data centers, we conduct extensive experiments using state-of-the-art deep network architectures on different types of data. Results demonstrate the effectiveness and robustness of the proposed method.
ISSN:1674-733X
1869-1919
DOI:10.1007/s11432-019-2705-2