Back to results
US 11,907,221 B2Granted
Scheduling parallel execution of query sub-plans
AssigneeSnowflake Inc.
InventorsSebastian Breß, Moritz Eyssen, Max Heimel
Life of this patent
Estimating the term…
Abstract
Sub-plans are executed in parallel using a plurality of execution nodes, which can be part of a data platform. In particular, various embodiments identify sub-plans (e.g., fragments or portions of one or more child operators) of a root operator in a query plan such that the identified sub-plans that are candidates for execution on a single execution node, determine a cost estimate for causing the candidate sub-plans to be executed in parallel using multiple execution nodes, and cause the candidate sub-plans to be executed in parallel based on the cost estimate.
Claims30
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
- accessing a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes;
- comparing the first cost to the second cost;
- determining, based on the comparing, whether the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- and
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes by performing, for each single sub-plan in the sub-plurality of sub-plans, a single scheduling operation to assign execution of an entire portion of the single sub-plan to a single execution node of the plurality of execution nodes, the single execution node being configured to generate a local schedule queue to process the single sub-plan assigned to the single execution node.
8 more elements
A method comprising:
- accessing a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes;
- comparing the first cost to the second cost;
- determining, based on the comparing, whether the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- and
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes by performing, for each single sub-plan in the sub-plurality of sub-plans, a single scheduling operation to assign execution of an entire portion of the single sub-plan to a single execution node of the plurality of execution nodes, the single execution node being configured to generate a local schedule queue to process the single sub-plan assigned to the single execution node.
5 more elements
A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
- accessing a query plan that comprises at least one root operator, the at least one root operator being associated with a plurality of sub-plans;
- identifying, from the plurality of sub-plans, a sub-plurality of sub-plans that each involves processing a data set that satisfies a set of criteria;
- estimating a first cost for executing individual sub-plans of the sub-plurality of sub-plans successively using a plurality of execution nodes;
- estimating a second cost for executing the sub-plurality of sub-plans in parallel using the plurality of execution nodes;
- comparing the first cost to the second cost;
- determining, based on the comparing, whether the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes;
- and
- in response to determining that the sub-plurality of sub-plans is to be executed in parallel using the plurality of execution nodes, causing the sub-plurality of sub-plans to be executed in parallel using the plurality of execution nodes by performing, for each single sub-plan in the sub-plurality of sub-plans, a single scheduling operation to assign execution of an entire portion of the single sub-plan to a single execution node of the plurality of execution nodes, the single execution node being configured to generate a local schedule queue to process the single sub-plan assigned to the single execution node.
5 more elements
Dependencies are inferred from the claim text.