Reinforcement learning based energy efficient resource allocation strategy of MapReduce jobs with deadline constraint

Big Data applications require more energy consumption to process a massive volume of data in a heterogeneous environment. Moreover, reducing energy consumption in Big Data applications is an important research topic. It is one of the challenging issues to conserve energy with a deadline constraint i...

Full description

Saved in:
Bibliographic Details
Published inCluster computing Vol. 26; no. 5; pp. 2719 - 2735
Main Author Lingam, Greeshma
Format Journal Article
LanguageEnglish
Published New York Springer US 01.10.2023
Springer Nature B.V
Subjects
Algorithms
Batch processing
Big Data
Clustering
Computer Communication Networks
Computer Science
Deadlines
Employment
Energy conservation
Energy consumption
Experimentation
Operating Systems
Optimization
Processor Architectures
Resource allocation
Scheduling
Workloads
Resource scheduling
Big data
Learning automata
MapReduce
Online AccessGet full text

Cover

More Information
Summary:Big Data applications require more energy consumption to process a massive volume of data in a heterogeneous environment. Moreover, reducing energy consumption in Big Data applications is an important research topic. It is one of the challenging issues to conserve energy with a deadline constraint in a heterogeneous environment. In this paper, we formulate scheduling the MapReduce jobs as a minimization problem by considering the decision variables with a user-specified deadline constraint. Further, a Learning Automata-based MapReduce Scheduling (LA-MRS) algorithm has been proposed to identify the resource allocation and save energy consumption of MapReduce tasks in a heterogeneous environment. We perform experimentation on the proposed LA-MRS algorithm using Hibench benchmark workloads such as Enhanced DFSIO, Nutch Indexing, k-mean Clustering and Hive Join. The experimentation illustrates that the proposed LA-MRS algorithm schedules the MapReduce task by saving around 25% of less energy consumed when compared to the existing algorithms.
ISSN:1386-7857
1573-7543
DOI:10.1007/s10586-022-03761-6