Empirical Study of Job Scheduling Algorithms in Hadoop MapReduce

• Published in: Cybernetics and information technologies : CIT Vol. 17; no. 1; pp. 146 - 163
• Main Authors: Gautam, Jyoti V., Prajapati, Harshadkumar B., Dabhi, Vipul K., Chaudhary, Sanjay
• Format: Journal Article
• Language: English
• Published: De Gruyter Open 01.03.2017; Sciendo
• Subjects: analysis; Big Data; experimental evaluation; Hadoop; job scheduling; Map Reduce
• ISSN: 1314-4081; 1314-4081
• DOI: 10.1515/cait-2017-0012

Several Job scheduling algorithms have been developed for Hadoop-Map Reduce model, which vary widely in design and behavior for handling different issues such as locality of data, user share fairness, and resource awareness. This article focuses on empirically evaluating the performance of three schedulers: First In First Out (FIFO), Fair scheduler, and Capacity scheduler. To carry out the experimental evaluation, we implement our own Hadoop cluster testbed, consisting of four machines, in which one of the machines works as the master node and all four machines work as slave nodes. The experiments include variation in data sizes, use of two different data processing applications, and variation in the number of nodes used in processing. The article analyzes the performance of the job scheduling algorithms based on various relevant performance measures. The results of the experiments are evident of the performance being affected by the job scheduling parameters, the type of applications, the number of nodes in the cluster, and size of the input data.