An effective algorithm for parallelizing sort merge joins in the presence of data skew

• Published in: Databases in Parallel and Distributed Systems: 2nd International Symposium pp. 103 - 115
• Main Authors: Wolf, Joel L., Dias, Daniel M., Yu, Philip S.
• Format: Conference Proceeding
• Language: English
• Published: New York, NY, USA ACM 01.07.1990
• Series: ACM Conferences
• Subjects: Information systems > Data management systems > Database design and models; Information systems > Data management systems > Query languages
• ISBN: 9780818620522; 0818620528
• DOI: 10.1145/319057.319072

Parallel processing of relational queries has received considerable attention of late. However, in the presence of data skew, the speedup from conventional parallel join algorithms can be very limited, due to load imbalances among the various processors. Even a single large skew element can cause a processor to become overloaded. In this paper, we propose a parallel sort merge join algorithm which uses a divide-and-conquer approach to address the data skew problem. The proposed algorithm adds an extra scheduling phase to the usual sort, transfer and join phases. During the scheduling phase, a parallelizable optimization algorithm, using the output of the sort phase, attempts to balance the load across the multiple processors in the subsequent join phase. The algorithm naturally identifies the largest skew elements, and assigns each of them to an optimal number of processors. Assuming a Zipf-like distribution for data skew, the algorithm is demonstrated to achieve very good load balancing for the join phase in a CPU-bound environment, and is shown to be very robust relative to the degree of data skew and the total number of processors.