Back to results
US 11,379,480 B1Granted
Parallel execution of query sub-plans
AssigneeSnowflake Inc.
InventorsSebastian Breß, Moritz Eyssen, Max Heimel
Life of this patent
Estimating the term…
Abstract
Sub-plans are executed in parallel using a plurality of execution nodes, which can be part of a data platform. In particular, sub-plans (e.g., fragments or portions of one or more child operators) of a root operator are identified in a query plan such that the identified sub-plans that are candidates for execution on a single execution node, determine a cost estimate for causing the candidate sub-plans to be executed in parallel using multiple execution nodes, and cause the candidate sub-plans to be executed in parallel based on the cost estimate.
Claims28
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
- accessing a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes;
- comparing the first cost to the second cost;
- determining, based on the comparing, whether the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes;
- and
- in response to determining that the sub-plurality of sub-plans is not to be executed in parallel using the plurality of execution nodes, causing the individual sub-plans of the sub-plurality of sub-plans to be executed successively such that all execution nodes of the plurality of execution nodes are reserved for execution of the individual sub-plan until the individual sub-plan completes execution.
9 more elements
A method comprising:
- accessing, by a hardware processor, a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, by the hardware processor and from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating, by the hardware processor, a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating, by the hardware processor, a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes;
- comparing, by the hardware processor, the first cost to the second cost;
- determining, by the hardware processor and based on the comparing, that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing, by the hardware processor, the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes;
- and
- executed in parallel using the plurality of execution nodes, causing the individual sub-plans of the sub-plurality of sub-plans to be executed successively such that all execution nodes of the plurality of execution nodes are reserved for execution of the individual sub-plan until the individual sub-plan completes execution.
6 more elements
A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
- accessing a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes using the plurality of execution nodes;
- comparing the first cost to the second cost;
- determining, based on the comparing, whether the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes;
- and
- in response to determining that the sub-plurality of sub-plans is not to be plans of the sub-plurality of sub-plans to be executed successively such that all individual sub-plan until the individual sub-plan completes execution.
6 more elements
Dependencies are inferred from the claim text.