MOMTH: multi-objective scheduling algorithm of many tasks in Hadoop

• Published in: Cluster computing Vol. 18; no. 3; pp. 1011 - 1024
• Main Authors: Nita, Mihaela-Catalina, Pop, Florin, Voicu, Cristiana, Dobre, Ciprian, Xhafa, Fatos
• Format: Journal Article Publication
• Language: English
• Published: New York Springer US 01.09.2015; Springer Nature B.V; Kluwer Academic Publishers
• Subjects: Algorithms; Apache Hadoop; Apache Hadoop (Programes d'ordinador); Applications programs; Big Data; Budgets; Cloud computing; Collaboration; Computació en núvol; Computer Communication Networks; Computer Science; data centers; Data processing; Employment; Energy consumption; Hadoop; Information retrieval; Informàtica; Linear programming; Llenguatges de programació; Macrodades; many task computing; MapReduce; Multiple objective analysis; multiprocessor tasks; networks; Operating Systems; Processor Architectures; Programació; Programming languages (Electronic computers); Response time; Scheduling; Software; Task scheduling; Workloads; Àrees temàtiques de la UPC; Cloud computing; 68U20; Hadoop; Task scheduling; Many task computing; Big data; 68M20; 68M14; MapReduce
• ISSN: 1386-7857; 1573-7543
• DOI: 10.1007/s10586-015-0454-8

A real challenge sits in front of the business solutions these days, in the context of the big amount of data generated by complex software applications: efficiently using the given limited resources to accomplish specific operations and tasks. Depending on the type of application dealing with, when trying to deliver a certain service in a specific time and with a limited budget, a sequential application may be redesigned in a convenient way so that it will become scalable and able to run on multiple resources. Many task computing model brings together loosely coupled applications, composed of many dependent/independent tasks, which will work together for a common result. When asking for a certain service, the most frequently constraints addressed by the user are deadline and budget. This paper elaborates on a multi-objective scheduling algorithm of many tasks in Hadoop for big data processing, named MOMTH. We consider objective functions related to users and resources in the same time with constraints like deadline (scheduling in due time) and budget. The algorithm evaluation was realized in scheduling load simulator, a tool integrated in Hadoop. MobiWay, a collaboration platform that expose interoperability between a large number of sensing mobile devices and a wide-range of mobility applications, was chosen for performance analysis of MOMTH. We compared the proposed algorithm with first in first out and fair schedulers and we obtained similar performance for our approach.