HeterPS: Distributed Deep Learning With Reinforcement Learning Based Scheduling in Heterogeneous Environments

• Published in: arXiv.org
• Main Authors: Liu, Ji, Wu, Zhihua, Yu, Dianhai, Ma, Yanjun, Feng, Danlei, Zhang, Minxu, Wu, Xinxuan, Yao, Xuefeng, Dou, Dejing
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 07.06.2023
• Subjects: Artificial neural networks; Computer networks; Data communication; Data storage; Deep learning; Distributed processing; Machine learning; Mathematical models; Parameters; Scheduling; Training; Workload

Deep neural networks (DNNs) exploit many layers and a large number of parameters to achieve excellent performance. The training process of DNN models generally handles large-scale input data with many sparse features, which incurs high Input/Output (IO) cost, while some layers are compute-intensive....