New algorithms for join and grouping operations

• Published in: Computer science (Berlin, Germany) Vol. 27; no. 1; pp. 3 - 27
• Main Author: Graefe, Goetz
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer-Verlag 01.02.2012
• Subjects: Computer Hardware; Computer Science; Computer Systems Organization and Communication Networks; Data Structures and Information Theory; Software Engineering/Programming and Operating Systems; Special Issue Paper; Theory of Computation; Intersection; Outer join; Buffer pool; Duplicate removal; Merging; Relational databases; Mistake; Difference; Union; Equivalence classes; Robust performance; Order-preserving; Map-reduce; Query optimization choice; Query execution algorithms; Uniqueness; Error; Grouping; Sort order; SQL; Semi-join; Query processing; Interesting orderings; Inner join; Run generation; Group by
• ISSN: 1865-2034; 1865-2042
• DOI: 10.1007/s00450-011-0186-9

Traditional database query processing relies on three types of algorithms for join and for grouping operations. For joins, index nested loops join exploits an index on its inner input, merge join exploits sorted inputs, and hash join exploits differences in the sizes of the join inputs. For grouping, an index-based algorithm has been used in the past whereas today sort- and hash-based algorithms prevail. Cost-based query optimization chooses the most appropriate algorithm for each query and for each operation. Unfortunately, mistaken algorithm choices during compile-time query optimization are common yet expensive to investigate and to resolve. Our goal is to end mistaken choices among join algorithms and among grouping algorithms by replacing the three traditional types of algorithms with a single one. Like merge join, this new join algorithm exploits sorted inputs. Like hash join, it exploits different input sizes for unsorted inputs. In fact, for unsorted inputs, the cost functions for recursive hash join and for hybrid hash join have guided our search for the new join algorithm. In consequence, the new join algorithm can replace both merge join and hash join in a database management system. The in-memory components of the new join algorithm employ indexes. If the database contains indexes for one (or both) of the inputs, the new join can exploit persistent indexes instead of temporary in-memory indexes. Using database indexes to find matching input records, the new join algorithm can also replace index nested loops join. In addition to join operations, a very similar algorithm supports grouping (“group by” queries in SQL) and duplicate elimination. For unsorted inputs, candidate output records take on the role of one of the inputs in a join operation. Our goal is to define a single grouping algorithm that can replace grouping by repeated index searches, by sorting, and by hashing. In other words, our goal is to end mistaken algorithm choices not only for joins and other binary matching operations but also for grouping and other unary matching operations in database query processing. Finally, these new algorithms can be instrumental for efficient and robust data processing in a map-reduce environment, because ‘map’ and ‘reduce’ operations are similar in essentials to join and grouping operations. Results from an implementation of the core algorithm are reported.