Query acceleration and metadata caching
AssigneeGoogle Inc.
InventorsThibaud Hottelier, Anoop Kochummen Johnson, Justin Levandoski, Deepak Choudhary Nettem, Yuri Volobuev
Life of this patent
Estimating the term…
Abstract
Aspects of the disclosure are directed to a metadata cache for extending data warehouse features to data lakes. The metadata cache can accelerate query execution by directly accessing unmanaged data from the data lake rather than accessing the data through the data warehouse. The metadata cache can allow for filtering the unmanaged data to improve the speed of retrieving data for executing a query.
Claims20
A method for processing queries, comprising:
- receiving, by one or more processors, a request from a query engine to access unmanaged data from a plurality of cloud storage tables of a data lake for processing a query;
- accessing, by the one or more processors, information about the unmanaged data, the information comprising table states for the cloud storage tables and schema for the cloud storage tables;
- identifying, by the one or more processors, a first table and a second table of the cloud storage tables for processing the query based on the table states;
- inferring, by the one or more processors, that a first type of column is present in both the first table and the second table based on the schemas for the cloud storage tables;
- retrieving, by the one or more processors, data from the first type of column of the first table based on data in a second column of the first table matching a condition of the query;
- filtering, by the one or more processors, the second table based on the data retrieved from the first table to generate a filtered table;
- and
- retrieving, by the one or more processors, the unmanaged data from the filtered table for responding to the query.
A system comprising:
- one or more processors;
- and
- one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
- receiving a request from a query engine to access unmanaged data from a plurality of cloud storage tables of a data lake for processing a query;
- accessing information about the unmanaged data, the information comprising table states for the cloud storage tables and schema for the cloud storage tables;
- identifying a first table and a second table of the cloud storage tables for processing the query based on the table states;
- inferring that a first type of column is present in both the first table and the second table based on the schemas for the cloud storage tables;
- retrieving data from the first type of column of the first table based on data in a second column of the first table matching a condition of the query;
- filtering the second table based on the data retrieved from the first table to generate a filtered table;
- and
- retrieving the unmanaged data from the filtered table for responding to the query.
A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
- receiving a request from a query engine to access unmanaged data from a plurality of cloud storage tables of a data lake for processing a query;
- accessing information about the unmanaged data, the information comprising table states for the cloud storage tables and schema for the cloud storage tables;
- identifying a first table and a second table of the cloud storage tables for processing the query based on the table states;
- inferring that a first type of column is present in both the first table and the second table based on the schemas for the cloud storage tables;
- retrieving data from the first type of column of the first table based on data in a second column of the first table matching a condition of the query;
- filtering the second table based on the data retrieved from the first table to generate a filtered table;
- and
- retrieving the unmanaged data from the filtered table for responding to the query.
Dependencies are inferred from the claim text.