GraphScope Flex: LEGO-like Graph Computing Stack

• Published in: arXiv.org
• Main Authors: He, Tao, Hu, Shuxian, Lai, Longbin, Li, Dongze, Li, Neng, Li, Xue, Liu, Lexiao, Luo, Xiaojian, Lyu, Binqing, Meng, Ke, Shen, Sijie, Li, Su, Wang, Lei, Xu, Jingbo, Yu, Wenyuan, Zeng, Weibin, Zhang, Lei, Zhang, Siyuan, Zhou, Jingren, Zhou, Xiaoli, Zhu, Diwen
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 19.12.2023
• Subjects: Computation; Computer Science - Databases; Computer Science - Distributed, Parallel, and Cluster Computing; Data storage; Iterative methods; Modularity; Social networks
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2312.12107

Graph computing has become increasingly crucial in processing large-scale graph data, with numerous systems developed for this purpose. Two years ago, we introduced GraphScope as a system addressing a wide array of graph computing needs, including graph traversal, analytics, and learning in one system. Since its inception, GraphScope has achieved significant technological advancements and gained widespread adoption across various industries. However, one key lesson from this journey has been understanding the limitations of a "one-size-fits-all" approach, especially when dealing with the diversity of programming interfaces, applications, and data storage formats in graph computing. In response to these challenges, we present GraphScope Flex, the next iteration of GraphScope. GraphScope Flex is designed to be both resource-efficient and cost-effective, while also providing flexibility and user-friendliness through its LEGO-like modularity. This paper explores the architectural innovations and fundamental design principles of GraphScope Flex, all of which are direct outcomes of the lessons learned during our ongoing development process. We validate the adaptability and efficiency of GraphScope Flex with extensive evaluations on synthetic and real-world datasets. The results show that GraphScope Flex achieves 2.4X throughput and up to 55.7X speedup over other systems on the LDBC Social Network and Graphalytics benchmarks, respectively. Furthermore, GraphScope Flex accomplishes up to a 2,400X performance gain in real-world applications, demonstrating its proficiency across a wide range of graph computing scenarios with increased effectiveness.