GraphMP: An Efficient Semi-External-Memory Big Graph Processing System on a Single Machine

• Published in: 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS) pp. 276 - 283
• Main Authors: Sun, Peng, Wen, Yonggang, Duong, Ta Nguyen Binh, Xiao, Xiaokui
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2017
• Subjects: Big Data; Computational modeling; Data models; Graph Processing; Load modeling; Memory management; Microsoft Windows; Parallel Computing; Servers; Venus
• ISSN: 1521-9097; 2690-5965
• DOI: 10.1109/ICPADS.2017.00045

Recent studies showed that single-machine graph processing systems can be as highly competitive as clusterbased approaches on large-scale problems. While several out-of-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform state-of-the-art systems such as GraphChi, X-Stream and GridGraph by 31.6x, 54.5x and 23.1x respectively, when running popular graph applications on a billion-vertex graph.