IoTDeM: An IoT Big Data-oriented MapReduce performance prediction extended model in multiple edge clouds

• Published in: Journal of parallel and distributed computing Vol. 118; pp. 316 - 327
• Main Authors: Lu, Zhihui, Wang, Nini, Wu, Jie, Qiu, Meikang
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.08.2018
• Subjects: Big Data; Edge cloud; Hadoop; IoT; Job estimation; MapReduce; Performance modeling; Performance prediction; Job estimation; Edge cloud; Hadoop; Big Data; Performance modeling; Performance prediction; IoT; MapReduce
• ISSN: 0743-7315; 1096-0848
• DOI: 10.1016/j.jpdc.2017.11.001

Uploading all IoT Big Data to a centralized cloud for data analytics is infeasible because of the excessive latency and bandwidth limitation of the Internet. A promising approach to addressing the challenges for data analytics in IoT is “edge cloud” that pushes various computing and data analysis capabilities to multiple edge clouds. MapReduce provides an efficient way to deal with a large amount of data. When performing data analysis, a challenge is to predict the performance of MapReduce jobs. In this paper, we propose and evaluate IoTDeM, which is an extended IoT Big Data-oriented model for predicting MapReduce performance in multiple edge clouds. IoTDeM is able to predict MapReduce jobs’ total execution time in a general implementation scenario with varying reduce amounts and cluster scales in Hadoop 2, rather than Hadoop 1. The extended model is based on historical job execution records and Locally Weighted Linear Regression (LWLR) techniques to predict the execution time of each job. Through extracting more representative features to represent a job, the IoTDeM model selects a cluster scale as a crucial parameter to further extend LWLR model. In the environment of Hadoop 2 with Ceph as the storage system, the experiments verify IoTDeM can effectively predict the total execution time of MapReduce applications, with the average relative error of less than 10%. •An IoT big data edge cloud architecture to address the challenges of data analytics in IoT using “edge cloud” that pushes various computing and data analysis capabilities to multiple edge clouds.•We propose IoTDeM, an extended IoT big data-oriented model for predicting MapReduce performances through extending the LWLR model in multiple Edge Clouds Hadoop 2 environments. By extracting distinguishing features for job representations, IoTDeM selects a cluster scale as a crucial parameter to improve the LWLR prediction model used in Hadoop 1 to adapt to Hadoop 2.•We propose our IoT Big data Edge Cloud Architecture based on Ceph, even though many previous researches on MapReduce performance prediction are mostly based on HDFS. Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.•We have validated the accuracy of the proposed MapReduce performance model using TestDFSIO and Sort benchmark applications in the IoT Big data Edge Cloud Architecture environment based on Hadoop 2 with Ceph as a storage system.