Balanced content space partitioning for pub/sub: a study on impact of varying partitioning granularity

• Published in: The Journal of supercomputing Vol. 77; no. 12; pp. 13676 - 13702
• Main Authors: Yoon, Daegun, Li, Zhetao, Oh, Sangyoon
• Format: Journal Article
• Language: English
• Published: New York Springer US 2021; Springer Nature B.V
• Subjects: Advances in Big Data and Deep Learning; Compilers; Computer Science; Correlation coefficients; Empirical analysis; Evaluation; Interpreters; Load balancing; Message processing; Partitioning; Processor Architectures; Programming Languages; Skewed distributions; Subscriptions; Workload; Workloads; Load balance; Content space partitioning; Publish/subscribe system; Partitioning granularity
• ISSN: 0920-8542; 1573-0484
• DOI: 10.1007/s11227-021-03821-5

The increasing volume of live content poses new challenges to publish/subscribe services at the cloud scale. Providing efficient publish/subscribe services for live content is a complex task because most subscriptions occupy only a small portion of the entire subscription space, i.e., use limited live content. Thus, the real-world workload of a publish/subscribe service for live content becomes skewed, and the distribution of subscriptions becomes seriously imbalanced, causing an inefficient processing of events. We present a correlation-based balanced content space partitioning technique for a publish/subscribe service. Our proposed technique reduces the degree of imbalance from a skewed subscription workload in a content-based publish/subscribe service, using the correlation coefficient between attributes to build dimension groups. We assign attributes of low correlation to the same dimension group to balance the subscription workloads. Moreover, we present our analysis on the load balance impacts of varying partitioning granularity for efficient message processing. We conducted empirical experiments evaluating the effectiveness of our partitioning technique and measuring the impact of varying partitioning granularity. The results show that the proposed technique outperforms conventional partitioning techniques by evaluating the ways in which subscriptions are evenly distributed among brokers. Moreover, the results show that the load balance can be improved by increasing the partitioning granularity with an adjustment of two degrees, i.e., the segment and dimension group degrees.