A Run Time Technique for Handling Error in User-Estimated Execution Times on Systems Processing MapReduce Jobs with Deadlines

• Published in: 2017 IEEE 5th International Conference on Future Internet of Things and Cloud (FiCloud) pp. 1 - 9
• Main Authors: Lim, Norman, Majumdar, Shikharesh, Ashwood-Smith, Peter
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.08.2017
• Subjects: Cloud computing; Handling error in user-estimated job/task execution times; MapReduce with SLAs; Processor scheduling; Resource allocation and scheduling; Resource management; Runtime; Scheduling; System performance
• DOI: 10.1109/FiCloud.2017.32

Effective management of resources on a cloud or cluster is crucial for achieving the quality of service requirements of users, which are typically captured in service level agreements (SLAs). This paper focuses on improving the robustness of resource allocation and scheduling techniques that process an open stream of MapReduce jobs with SLAs, by introducing techniques to handle errors/inaccuracies in user-estimated execution times that are submitted as part of the job's SLA. Inaccuracies in the estimates of task execution times can prevent the resource allocation and scheduling algorithm from making effective scheduling decisions, leading to a degradation in system performance. Techniques for handling error during runtime are presented to handle the situation where jobs have already started executing and their estimated execution times are inaccurate. A simulation-based performance evaluation of the error handling techniques is conducted, which demonstrates that the techniques are effective in improving system performance.