Fast Coflow Scheduling via Traffic Compression and Stage Pipelining in Datacenter Networks

• Published in: IEEE Transactions on Computers Vol. 68; no. 12; pp. 1755 - 1771
• Main Authors: Zhou, Qihua, Wang, Kun, Li, Peng, Zeng, Deze, Guo, Song, Ye, Baoliu, Guo, Minyi
• Format: Journal Article
• Language: English; Japanese
• Published: New York IEEE 01.12.2019; Institute of Electrical and Electronics Engineers (IEEE); The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Analytics; Bandwidth; Cluster frameworks; Coflow scheduling; Completion time; Compression; Computer simulation; Data centers; Data compression; Datacenter networks; Heuristic methods; Job shop scheduling; Measurement; Pipeline processing; Pipelining; Prediction algorithms; Processor scheduling; Production scheduling; Recurrent neural networks; Resource conservation; Resource utilization; Task scheduling
• ISSN: 0018-9340; 2326-3814; 1557-9956
• DOI: 10.1109/TC.2019.2931716

Big data analytics in datacenters often involve scheduling of data-parallel jobs. Traditional scheduling techniques based on improving network resource utilization are subject to limited bandwidth in datacenter networks. To alleviate the shortage of bandwidth, some cluster frameworks employ techniques of traffic compression to reduce transmission consumption. However, they tackle scheduling in a coarse-grained manner at task level and do not perform well in terms of flow-level metrics due to high complexity. Fortunately, the abstraction of coflow pioneers a new perspective to facilitate scheduling efficiency. In this paper, we introduce a coflow compression mechanism to minimize the completion time in data-intensive applications. Due to the NP-hardness, we propose a heuristic algorithm called Fastest-Volume-Disposal-First (FVDF) to solve this problem. For online applicability, FVDF supports stage pipelining to accelerate scheduling and exploits recurrent neural networks (RNNs) to predict compression speed. Meanwhile, we build Swallow, an efficient scheduling system that implements our proposed algorithms. It minimizes coflow completion time (CCT) while guaranteeing resource conservation and starvation freedom. The results of both trace-driven simulations and real experiments show the superiority of our algorithm, over existing one. Specifically, Swallow speeds up CCT and job completion time (JCT) by up to 1.47χ and 1.66χ on average, respectively, over the SEBF in Varys, one of the most efficient coflow scheduling algorithms so far. Moreover, with coflow compression, Swallow reduces data traffic by up to 48.41 percent on average.