Fast Coflow Scheduling via Traffic Compression and Stage Pipelining in Datacenter Networks

• Published in: IEEE Transactions on Computers Vol. 68; no. 12; pp. 1755 - 1771
• Main Authors: Zhou, Qihua, Wang, Kun, Li, Peng, Zeng, Deze, Guo, Song, Ye, Baoliu, Guo, Minyi
• Format: Journal Article
• Language: English; Japanese
• Published: New York IEEE 01.12.2019; Institute of Electrical and Electronics Engineers (IEEE); The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Analytics; Bandwidth; Cluster frameworks; Coflow scheduling; Completion time; Compression; Computer simulation; Data centers; Data compression; Datacenter networks; Heuristic methods; Job shop scheduling; Measurement; Pipeline processing; Pipelining; Prediction algorithms; Processor scheduling; Production scheduling; Recurrent neural networks; Resource conservation; Resource utilization; Task scheduling

Big data analytics in datacenters often involve scheduling of data-parallel jobs. Traditional scheduling techniques based on improving network resource utilization are subject to limited bandwidth in datacenter networks. To alleviate the shortage of bandwidth, some cluster frameworks employ techniqu...