US 12,754,911 B1Granted

Cross-format metadata for querying external tables

AssigneeAmazon Technologies, Inc.

InventorsEnrico Siragusa, Gaurav Saxena, Ippokratis Pandis, Fabian Oliver Nagel, Gerard Louw, Abhishek Sharma, Mario Guerriero

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Metadata for external tables stored as a table a database system may be evaluated to perform queries that access external tables stored in different formats. When a query is received, a query plan is generated that may include operations to evaluate a table that stores metadata for an external table according to a read view to identify data objects storing data of different portions of the external table to access and to access the identified data objects of the external table according information in the table stores the metadata. The plan may then be performed to generate a result of the query.

Claims20

Dependent claims

A system, comprising:

at least one processor;
and
a memory, storing program instructions that when executed by the at least one processor, cause the at least one processor to implement a database system;
wherein the database system is configured to:
receive a query that accesses an external table stored separately from the database system, wherein the database system supports external tables stored in different file formats, including a transactional file format and a non-transaction file format, wherein the external table is stored in one of the different file formats;
generate a plan to perform the query, wherein the plan comprises:
a first operation to evaluate an additional table that stores metadata that describes different files that make up the external table according to the one of the different file formats;
and
a second operation to access the identified one of more files using respective storage locations of the different files stored in the additional table;
and
perform the plan to generate a result of the query, wherein to perform the query plan, the database system is configured to:
cause one or more compute nodes, different from a leader node that generated the query plan, to execute the first operation to access the metadata and identify one or more files of the different files of the external table to access according to a read view determined for the query;
and
cause the one or more compute nodes to execute the second operation to access the identified one of more files using respective storage locations of the different files stored in the additional table;
and
return the result of the query.

13 more elements

A method, comprising:

receiving, at a database system, a query that causes access to an external table stored separately from the database system;
generating, by the database system, a plan to perform the query, wherein generating the plan comprises:
including a first operation to evaluate an additional table that stores metadata that comprises respective information to access different data objects storing respective data of different portions of the external table;
and
including a second operation to access the identified one of more data objects using the respective information in the additional table;
and
performing, by the database system, the plan to generate a result of the query, comprising:
causing one or more nodes, different from a node that generated the query plan, to execute the first operation to access the metadata and identify one or more files of the different files of the external table to access according to a read view determined for the query;
and
causing the one or more nodes to execute the second operation to access the identified one of more files using respective storage locations of the different files stored in the additional table.

7 more elements

One or more non-transitory, computer-readable storage media, storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement:

receiving, at a database system, a query that causes access to an external table stored separately from the database system;
generating, by the database system, a plan to perform the query, wherein the plan comprises:
a first operation to evaluate an additional table that stores metadata that comprises respective information to access different data objects storing respective data of different portions of the external table;
and
a second operation to access the identified one of more data objects using the respective information in the additional table;
and
performing, by the database system, the plan to generate a result of the query, comprising:
causing one or more nodes, different from a node that generated the query plan, to execute the first operation to access the metadata and identify one or more files of the different files of the external table to access according to a read view determined for the query;
and
causing the one or more nodes to execute the second operation to access the identified one of more files using respective storage locations of the different files stored in the additional table;
and
returning, by the database system, the result of the query.

9 more elements

Dependencies are inferred from the claim text.