Submodularity of Distributed Join Computation

• Published in: Proceedings - ACM-SIGMOD International Conference on Management of Data Vol. 2018; p. 1237
• Main Authors: Li, Rundong, Riedewald, Mirek, Deng, Xinyan
• Format: Journal Article
• Language: English
• Published: United States 01.06.2018
• Subjects: Information systems → Parallel and distributed DBMSs; load balancing; Distributed join; MapReduce algorithms; Computing methodologies → Distributed algorithms; load variance minimization; Theory of computation → Distributed algorithms
• ISSN: 0730-8078
• DOI: 10.1145/3183713.3183728

We study distributed equi-join computation in the presence of join-attribute skew, which causes load imbalance. Skew can be addressed by more fine-grained partitioning, at the cost of input duplication. For random load assignment, e.g., using a hash function, fine-grained partitioning creates a tradeoff between load expectation and variance. We show that minimizing load variance subject to a constraint on expectation is a monotone submodular maximization problem with Knapsack constraints, hence admitting provably near-optimal greedy solutions. In contrast to previous work on formal optimality guarantees, we can prove this result also for self-joins and more general load functions defined as weighted sum of input and output. We further demonstrate through experiments that this theoretical result leads to an effective algorithm for the problem of minimizing running time, even when load is assigned deterministically.