Cost and fault-tolerant aware resource management for scientific workflows using hybrid instances on clouds

• Published in: Multimedia tools and applications Vol. 77; no. 8; pp. 10171 - 10193
• Main Authors: K., Vinay, Kumar, S. M. Dilip, S., Raghavendra, R., Venugopal K.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2018; Springer Nature B.V
• Subjects: Algorithms; Bids; Cloud computing; Computer Communication Networks; Computer networks; Computer Science; Computer simulation; Data analysis; Data Structures and Information Theory; Empirical analysis; Fault tolerance; Multimedia Information Systems; Provisioning; Resource allocation; Resource management; Scheduling; Scheduling algorithms; Special Purpose and Application-Based Systems; Volatility; Instances; Cloud computing; Spot; Fault-tolerant; On-demand; Scientific workflows; Scheduling; Blockspot
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-017-5304-7

Cloud service providers are offering computing resources at a reasonable price as a pay-per-use model. Further, cloud service providers have also introduced different pricing models like spot, blockspot and spotfleet instances that are cost effective and user’s have to go through the bidding to balance the reliability and monetary costs. Henceforth, Scientific Workflows (SWf) that are used to model applications of high throughput, computation and complex large-scale data analysis are significantly adopting these computing resources. Nevertheless, spot instances are terminated when the market spot price exceeds the users bid price. Moreover, failures are inevitable in such a large distributed systems and often pose a challenge to design a fault-tolerant scheduling algorithm for SWf. This paper presents an efficient, low-cost and fault-tolerant scheduling algorithm and a bidding strategy to minimize the volatility and cost of resource provisioning for SWf. The proposed algorithm uses spot and blockspot instances as hybrid instances in comparison with on-demand instance to reduce the execution cost and fault-tolerant while meeting the SWf deadline. The results obtained reveal the promising potential of the proposed scheduling algorithm and are demonstrated through empirical simulation study that is robust under short deadlines with minimal makespan and cost.