Adaptive distribution method for hash operations
AssigneeSnowflake Computing, Inc.
InventorsBenoit Dageville, Thierry Cruanes, Marcin Zukowski, Allison Waingold Lee, Philipp Thomas Unterbrunner
Life of this patent
Estimating the term…
Abstract
A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.
Claims21
A method for joining relations distributed over a plurality of processing nodes, the method comprising:
- receiving a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the desired join uses an equivalence operation, wherein the plurality of relations comprises a first relation and a second relation, and wherein the second relation is estimated to be smaller than the first relation;
- processing the second relation according to a build operation, wherein in response to the build operation an actual size of the second relation is known after the second relation being fully consumed;
- in response to the build operation, determining whether to join the plurality of relations via a broadcasting join or a re-partitioning join based on the actual size of the second relation, an estimated size of the first relation, and a cost metric;
- and
- performing the selected broadcasting join or re-partitioning join.
A system for joining relations that are distributed over a plurality of processing nodes performing, the system comprising non-transitory storage media storing computer executable instructions which, when executed by one or more processors, cause the system to:
- receive a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the desired join uses an equivalence operation, wherein the plurality of relations comprises a first relation and a second relation, and wherein the second relation is estimated to be smaller than the first relation;
- processing the second relation according to a build operation, wherein in response to the build operation an actual size of the second relation is known after the second relation being fully consumed;
- in response to the build operation, determine whether to join the plurality of relations via a broadcasting join or a re-partitioning join based on the actual size of the second relation, an estimated size of the first relation, and a cost metric;
- and joining of the first and second relations using the selected broadcasting join or repartitioning join.
An apparatus for joining relations that are distributed over a plurality of processing nodes, wherein the apparatus is configured to:
- receive a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the desired join uses an equivalence operation, wherein the plurality of relations comprises a first relation and a second relation, and wherein the second relation is estimated to be smaller than the first relation;
- process the second relation according to a build operation, wherein in response to the build operation an actual size of the second relation is known after the second relation being fully consumed;
- in response to the build operation, determine whether to join the plurality of relations via a broadcasting join or a re-partitioning join based on the actual size of the second relation, an estimated size of the first relation, and a cost metric;
- and
- joining of the first and second relations using the selected broadcasting join or repartitioning join.
A system for joining relations that are distributed over a plurality of processing nodes, the system comprising:
- means for receiving a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the desired join uses an equivalence operation, wherein the plurality of relations comprises a first relation and a second relation, and wherein the second relation is estimated to be smaller than the first relation;
- means for processing the second relation according to a build operation, wherein in response to the build operation an actual size of the second relation is known after being fully consumed;
- means for determining, in response to the build operation, whether to join the plurality of relations via a broadcasting join or a re-partitioning join based primarily on the actual size of the second relation, an estimated size of the first relation, and a cost metric;
- and
- means for joining of the first and second relations using the selected broadcasting join or re-partitioning join.
Dependencies are inferred from the claim text.