AdapCC: Making Collective Communication in Distributed Machine Learning Adaptive

• Published in: Proceedings of the International Conference on Distributed Computing Systems pp. 25 - 35
• Main Authors: Zhao, Xiaoyang, Zhang, Zhe, Wu, Chuan
• Format: Conference Proceeding
• Language: English
• Published: IEEE 23.07.2024
• Subjects: collective communication; distributed training; Graphics processing units; Libraries; Performance evaluation; Pressing; Rendering (computer graphics); Throughput; Training
• ISSN: 2575-8411
• DOI: 10.1109/ICDCS60910.2024.00012

As deep learning (DL) models continue to grow in size, there is a pressing need for distributed model learning using a large number of devices (e.g., G PU s) and servers. Collective communication among devices/servers (for gradient synchronization, intermediate data exchange, etc.) introduces significant overheads, rendering major performance bottlenecks in distributed learning. A number of communication libraries, such as NCCL, Gloo and MPI, have been developed to optimize collective communication. Predefined communication strategies (e.g., ring or tree) are largely adopted, which may not be efficient or adaptive enough for inter-machine communication, especially in cloud-based scenarios where instance configurations and network performance can vary substantially. We propose AdapCC, a novel communication library that dynamically adapts to resource heterogeneity and network variability for optimized communication and training performance. AdapCC generates communication strategies based on run-time profiling, mitigates resource waste in waiting for computation stragglers, and executes efficient data transfers among DL workers. Experimental results under various settings demonstrate 2x communication speed-up and 31 % training throughput improvement with AdapCC, as compared to NCCL and other representative communication backends.