US 7,984,043 B1Granted

System and method for distributed query processing using configuration-independent query plans

AssigneeAmazon Technologies, Inc.

InventorFlorian M. Waas

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system and method for distributed query processing may compile and optimize query plans for incoming query requests independent of hardware configurations and/or physical locations of data partitions in a distributed storage system (e.g., a data grid). The query plan may be divided into segments, and each segment may be instantiated on a remote query processing node of the distributed system by a query coordinator node according to metadata accessed at runtime by remote sub-query operators in the query plan. The metadata may include an indication of the physical locations of data partitions in the system and may be stored on one or more of the query processing nodes. The remote query processing nodes may execute the query plan segments and return results to the requestor. Cached query plans may be re-executed without recompilation, according to current metadata, even in the event of a node failure or data partition move.

Claims49

Dependent claims

A computing system, comprising:

a plurality of query processing nodes, each comprising:
a query processor;
and
a portion of a distributed storage system, wherein the distributed storage system comprises data stored on each of the plurality of query processing nodes;
wherein the query processor of one of the plurality of query processing nodes is configured to:
receive a query request;
generate a query plan for execution of the query request, wherein to generate the query plan, the query processor is configured to parse or translate the received query request, wherein the query plan generated by the query processor is not dependent on locations of data in the distributed storage system, wherein the query plan is generated such that the particular query processing nodes on which segments of the query plan are to be instantiated are not determined until the query plan is carried out;
and
when the query plan is carried out:
instantiate a segment of the query plan on each of one or more others of the plurality of query processing nodes, wherein instantiating comprises determining the query processing node on which each segment is to be instantiated;
receive query results from each of the one or more others of the plurality of query processing nodes;
and
return the query results.

10 more elements

A query processing node, comprising:

a query processor;
and
a portion of a distributed storage system, wherein the distributed storage system comprises data stored on each of a plurality of query processing nodes;
wherein the query processor is configured to:
receive a query request;
generate a query plan for execution of the query request, wherein to generate the query plan, the query processor is configured to parse or translate the received query request, wherein the query plan generated by the query processor is not dependent on a hardware configuration of the plurality of query processing nodes, wherein the query plan is generated such that the particular query processing nodes on which segments of the query plan are to be instantiated are not determined until the query plan is carried out;
and
when the query plan is carried out:
instantiate a segment of the query plan on each of one or more others of the plurality of query processing nodes, wherein instantiating comprises determining the query processing node on which each segment is to be instantiated;
receive query results from each of the one or more others of the plurality of query processing nodes;
and
return the query results.

9 more elements

Dependencies are inferred from the claim text.