Back to results
US 11,144,548 B2Granted
Optimized data structures of a relational cache with a learning capability for accelerating query execution by a data system
AssigneeDremio Corporation
InventorsTomer Shiran, Jacques Nadeau, Steven Michael Phillips
Life of this patent
Estimating the term…
Abstract
The disclosed embodiments include a method performed by a data system. The method includes automatically learning relationship(s) among datasets based on one or more of a user query or an observation of a data flow through the data system. The method further includes generating an optimized data structure based on the learned relationships among the datasets. The data system then modifies a query plan to obtain query results that satisfy a query by reading the optimized data structure in lieu of reading the datasets.
Claims26
A method performed by a data system, the method comprising:
- automatically learning one or more relationships among a plurality of datasets stored at one or more data sources based on one or more of a query received by the data system or an observation of a data flow of the data system;
- generating an optimized data structure based on the learned relationships among the plurality of datasets,
- wherein the optimized data structure includes a record-preserving join of the plurality of datasets;
- receiving a new query for execution in accordance with a query plan;
- in response to receiving the new query, processing a plurality of optimized data structures including the optimized data structure by:
- skipping to a next optimized data structure when a current record-preserving join lacks any physical dataset in common with the query plan, and
- pruning a physical dataset of a current record-preserving join when the physical dataset is not referred to in the query plan;
- and
- determining that modifying the query plan to utilize the pruned record-preserving join would accelerate execution of the new query compared to the execution of the query plan;
- and
- modifying the query plan to obtain query results that satisfy the query by reading the optimized data structure in lieu of reading the plurality of datasets.
8 more elements
A data system comprising:
- a processor;
- and
- memory containing instructions that, when executed by the processor, cause the data system to:
- automatically learn one or more relationships among a plurality of datasets based on one or more of a query received by the data system or an observation of a data flow of the data system, the plurality of datasets being stored at one or more data sources;
- generate an optimized data structure based on the learned relationships among the plurality of datasets, wherein the optimized data structure includes a record-preserving join of the plurality of datasets;
- store the optimized data structure in a cache storing a plurality of optimized data structures;
- receive a new query for execution in accordance with a query plan;
- in response to receiving the new query, process a plurality of optimized data structures including the optimized data structure by causing the data system to:
- skip to a next optimized data structure when a current record-preserving join lacks any physical dataset in common with the query plan, and
- prune a physical dataset of a current record-preserving join when the physical dataset is not referred to in the query plan;
- and
- determine that modifying the query plan to utilize the pruned record-preserving join would accelerate execution of the new query compared to the execution of the query plan;
- and
- modify a query plan to obtain query results that satisfy a query by reading the optimized data structure in lieu of reading the plurality of datasets stored at the one or more data sources.
11 more elements
A data system comprising:
- a processor;
- and
- memory containing instructions that, when executed by the processor, cause the data system to:
- identify a join formed of a fact table and a plurality of dimension tables stored at one or more data sources;
- determine that the join is a record-preserving join when it has only one record for each record of the fact table;
- generate an optimized data structure including the record-preserving join;
- store the optimized data structure in a cache memory storing a plurality of optimized data structures;
- and
- modify a query plan to obtain query results that satisfy a query by reading the optimized data structure in lieu of reading any of the fact table or the plurality of dimension tables stored at the one or more data sources.
6 more elements
A data system comprising:
- a cache memory configured to store a plurality of optimized data structures that each include a record-preserving join,
- wherein the record-preserving join is formed of a fact table and a plurality of dimension tables, and
- wherein the record-preserving join includes a record for each record of the fact table;
- and
- a processor configured to modify a query plan to obtain query results by reading at least some of the plurality of optimized data structures in lieu of reading data stored in a plurality of data sources.
2 more elements
Dependencies are inferred from the claim text.