Distributed query compilation and evaluation system and method
AssigneeIntellidimension, Inc.
InventorsGeoffrey Chappell, Derrish Repchick
Life of this patent
Estimating the term…
Abstract
A system and method for distributed query compilation and evaluation involving a first table having a first number of partitions and a second table having a second number of partitions. A query operation, such as a join, is performed for each combination of partitions from the first and second tables. The result of each query operation on each combination of partitions is organized into a third number of partitions. A number of tables equal to the third number are generated with each table including one of the partition results from each of the query operation combinations.
Claims10
A computer implemented method of evaluating a query including one or more query operations, the method comprising:
- obtaining a compiled query including a join operation against a first table, the first table having a first set of partitions, the first set including an X number of partitions, the query including a partitioning scheme for intermediate tables that includes an N number of partitions;
- obtaining a second set of partitions of a second table, the second set including a Y number of partitions;
- performing using a query engine the join operation of one or more partitions of the first set against one or more of the partitions of the second set to create a plurality of combination results, wherein if the join operation is a pairwise join, each of the partitions of the first set are joined against corresponding ones of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the pairwise join being the same as X and Y, wherein if the join operation is a crosswise join, each of the partitions of the first set is joined against each of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the crosswise join is X*Y;
- partitioning each of the plurality of combination results to produce a result set of result partitions for each of the plurality of combination results, each result set including N partitions;
- and
- reducing corresponding ones of the result partitions across the result sets to an intermediate table to produce a set of N intermediate tables, each intermediate table including data from a result partition from each of the result sets.
A machine readable hardware storage medium containing machine executable instructions for implementing a method of evaluating a query including one or more query operations, the instructions comprising:
- a set of instructions for obtaining a compiled query including a join operation against a first table, the first table having a first set of partitions, the first set including an X number of partitions, the query including a partitioning scheme for intermediate tables that includes an N number of partitions;
- a set of instructions for obtaining a second set of partitions of a second table, the second set including a Y number of partitions;
- a set of instructions for performing using a query engine the join operation of one or more partitions of the first set against one or more of the partitions of the second set to create a plurality of combination results, wherein if the join operation is a pairwise join, each of the partitions of the first set are joined against corresponding ones of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the pairwise join being the same as X and Y, wherein if the join operation is a crosswise join, each of the partitions of the first set is joined against each of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the crosswise join is X*Y;
- a set of instructions for partitioning each of the plurality of combination results to produce a result set of result partitions for each of the plurality of combination results, each result set including N partitions;
- and
- a set of instructions for reducing corresponding ones of the result partitions across the result sets to an intermediate table to produce a set of N intermediate tables, each intermediate table including data from a result partition from each of the result sets.
Dependencies are inferred from the claim text.