Back to results
US 12,216,656 B2Granted
Scalable query processing
AssigneeSnowflake Inc.
InventorsThierry Cruanes, Igor Demura, Varun Ganesh, Prasanna Rajaperumal, Libo Wang, Jiaqi Yan
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure may provide a dynamic query execution model. This query execution model may provide acceleration by scaling out parallel parts of a query (also referred to as a fragment) to additional computing resources, for example computing resources leased from a pool of computing resources. Execution of the parts of the query may be coordinated by a parent query coordinator, where the query originated, and a fragment query coordinator.
Claims21
A method comprising:
- receiving, by one or more processors, a query directed at a data set stored in a network-based database system;
- generating a query plan to execute the query;
- identifying a portion of the query plan that is eligible for fragment processing based on an output-to-input ratio where input represents size of one or more input files for fragment processing and output represents size of an output file based on the one or more input file;
- executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set to generate a first batch result;
- transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set to generate the output file using a second set of computing resources assigned to the fragment query coordinator, the output file being stored in a storage location;
- scanning, by the parent query coordinator, the output file from the storage location to generate a second batch result;
- combining the first batch result and the second batch result to generate combined results;
- and
- executing remaining portion of the query plan on the combined results to generate a response to the query.
6 more elements
A system comprising:
- one or more processors of a machine;
- and
- a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- receiving a query directed at a data set stored in a network-based database system;
- generating a query plan to execute the query;
- identifying a portion of the query plan that is eligible for fragment processing based on an output-to-input ratio where input represents size of one or more input files for fragment processing and output represents size of an output file based on the one or more input file;
- executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set to generate a first batch result;
- transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set to generate the output file using a second set of computing resources assigned to the fragment query coordinator, the output file being stored in a storage location;
- scanning, by the parent query coordinator, the output file from the storage location to generate a second batch result;
- combining the first batch result and the second batch result to generate combined results;
- and
- executing remaining portion of the query plan on the combined results to generate a response to the query.
9 more elements
A non-transitory machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- receiving a query directed at a data set stored in a network-based database system;
- generating a query plan to execute the query;
- identifying a portion of the query plan that is eligible for fragment processing based on an output-to-input ratio where input represents size of one or more input files for fragment processing and output represents size of an output file based on the one or more input file;
- executing, by a parent query coordinator using one or more computing resources of a first set of computing resources assigned to the parent query coordinator, the identified portion of the query plan on a first batch of files of the data set to generate a first batch result;
- transmitting instructions to a fragment query coordinator for the fragment query coordinator to execute the identified portion of the query on a second batch of files of the data set to generate the output file using a second set of computing resources assigned to the fragment query coordinator, the output file being stored in a storage location;
- scanning, by the parent query coordinator, the output file from the storage location to generate a second batch result;
- combining the first batch result and the second batch result to generate combined results;
- and
- executing remaining portion of the query plan on the combined results to generate a response to the query.
6 more elements
Dependencies are inferred from the claim text.