Handling Pregel's limits in big graphs processing in the presence of high degree vertices

Even if specialized distributed graph processing systems such as Pregel scale better than pure MapReduce programs, in graph processing, by reducing disk I/O for iterative algorithms while offering an easy programming model using "think like vertex" paradigm, large scale graphs processing i...

Full description

Saved in:
Bibliographic Details
Published inApplications of Big Data Analytics: Trends, Issues, and Challenges
Main Authors Al Hajj Hassan, Mohamad, Bamha, Mostafa
Format Book Chapter
LanguageEnglish
Published 29.05.2018
Subjects
Cognitive science
Computer science
Online AccessGet full text

Cover

More Information
Summary:Even if specialized distributed graph processing systems such as Pregel scale better than pure MapReduce programs, in graph processing, by reducing disk I/O for iterative algorithms while offering an easy programming model using "think like vertex" paradigm, large scale graphs processing is still challenging in the presence of high degree vertices: Communication and load imbalance among processing nodes can have disastrous effects on performance. In this article, we introduce a scalable MapReduce graph partitioning approach for high degree vertices using master/slave partitioning. This partitioning makes Pregel-like systems, in graph processing, scalable and insensitive to the effects of high degree vertices while guaranteeing perfect balancing properties of communication and computation during all the stages of big graphs processing. A cost model and performance analysis are given to show the effectiveness and the scalability of our graph partitioning approach in large scale systems.
ISBN:3319764713
9783319764719
DOI:10.1007/978-3-319-76472-6