AdapCC: Making Collective Communication in Distributed Machine Learning Adaptive

• Published in: Proceedings of the International Conference on Distributed Computing Systems pp. 25 - 35
• Main Authors: Zhao, Xiaoyang, Zhang, Zhe, Wu, Chuan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 23.07.2024
• Subjects: collective communication; distributed training; Graphics processing units; Libraries; Performance evaluation; Pressing; Rendering (computer graphics); Throughput; Training

As deep learning (DL) models continue to grow in size, there is a pressing need for distributed model learning using a large number of devices (e.g., G PU s) and servers. Collective communication among devices/servers (for gradient synchronization, intermediate data exchange, etc.) introduces signif...