US 11,537,610 B2Granted

Data statement chunking

AssigneeAtScale, Inc.

InventorsSarah Gerweck, David P. Ross, Daren Drummond

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques are presented for applying fine-grained client-specific rules to divide (e.g., chunk) data statements to achieve cost reduction and/or failure rate reduction associated with executing the data statements over a subject dataset. Data statements for the subject dataset are received from a client. Statement attributes derived from the data statements are processed with respect to fine-grained rules and/or other client-specific data to determine whether a data statement chunking scheme is to be applied to the data statements. If a data statement chunking scheme is to be applied, further analysis is performed to select a data statement chunking scheme. A set of data operations are generated based at least in part on the selected data statement chunking scheme. The data operations are issued for execution over the subject dataset. The results from the data operations are consolidated in accordance with the selected data statement chunking scheme and returned to the client.

Claims20

Dependent claims

A method for chunking data statements based at least in part on a set of client-specific data in a client data statement processing layer, the method comprising:

receiving a set of data statements issued by a client, the data statements issued by the client to operate over a subject dataset;
analyzing the set of data statements to determine statement attributes associated with the data statements;
applying a set of client-specific data to the statement attributes to determine a chunking scheme, the set of client-specific data including performance data, statement chunking rules, and a set of expanded dataset metadata, the set of expanded dataset metadata derived from client input and first dataset metadata, the first dataset metadata derived from the subject dataset, the statement chunking rules indicative of a dimension based chunking for data statements;
expanding the dataset metadata into a set of expanded dataset metadata;
consulting the expanded dataset metadata to perform at least one of determining the at least one chunking scheme, or generating one or more data operations;
based on consulting the expanded dataset metadata, generating a set of data operations from the set of data statements, the set of data operations generated based on the chunking scheme and one or more dimensions associated with the subject dataset based on the expanded dataset metadata;
accessing the performance data;
via the performance data, generating first performance estimates associated with the set of data statements;
applying the statement chunking rules to the first performance estimates to determine whether to invoke chunking on the set of data statements, and if so,
applying select statement chunking rules for accessing the set of expanded dataset metadata based on field specific attributes in the expanded dataset metadata to determine chunking parameters for chunking the set of data statements, the select statement chunking rules denoting the first performance estimates based on metrics derived from field specific performance, the select statements derived from the set of data statements;
generating a set of candidate chunking schemes from the chunking parameters;
accessing the performance data to generate second performance estimates for the set of candidate chunking schemes;
selecting the chunking scheme from the set of candidate chunking schemes based on the second performance estimates;
and
executing the set of data operations over the subject dataset to generate a result set.

12 more elements

A computer readable medium, embodied in a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by one or more processors causes the one or more processors to perform a set of acts for chunking data statements based at least in part on a set of client-specific information in a client data statement processing layer, the acts comprising:

receiving a set of data statements issued by a client, the data statements issued by the client to operate over a subject dataset;
analyzing the set of data statements to determine statement attributes associated with the data statements;
applying a set of client-specific data to the statement attributes to determine a chunking scheme, the set of client-specific data including performance data, statement chunking rules, and a set of expanded dataset metadata, the set of expanded dataset metadata derived from client input and first dataset metadata, the first dataset metadata derived from the subject dataset, the statement chunking rules indicative of a dimension based chunking for data statements;
receiving a set of dataset metadata associated with the subject dataset;
expanding the dataset metadata into a set of expanded dataset metadata;
consulting the expanded dataset metadata to perform at least one of, determining the at least one chunking scheme, or generating the one or more data operations;
based on consulting the expanded dataset metadata, generating data operations from the set of data statements, the data operations generated based on the chunking scheme and on one or more dimensions associated with the subject dataset based on the expanded dataset metadata;
accessing the performance data;
via the performance data, generating first performance estimates associated with the set of data statements;
applying the statement chunking rules to the first performance estimates to determine whether to invoke chunking on the set of data statements, and if so,
applying select statement chunking rules for accessing the set of expanded dataset metadata based on field specific attributes in the expanded dataset metadata to determine chunking parameters for chunking the set of data statements, the select statement chunking rules denoting the first performance estimates based on metrics derived from field specific performance, the select statements derived from the set of data statements;
generating a set of candidate chunking schemes from the chunking parameters;
accessing the performance data to generate second performance estimates for the set of candidate chunking schemes;
selecting the chunking scheme from the set of candidate chunking schemes based on the second performance estimates;
and
executing the data operations over the subject dataset to generate a result set.

13 more elements

A system for chunking data statements based at least in part on a set of client-specific information in a client data statement processing layer, the system comprising:

a storage medium having stored thereon a sequence of instructions;
and
one or more processors that execute the instructions to cause the one or more processors to perform a set of acts, the acts comprising:
receiving one or more data statements issued by at least one client, the data statements issued by the client to operate over a subject dataset;
applying at least a portion of a set of client-specific data to the data statements to determine at least one chunking scheme, the set of client-specific data including performance data, statement chunking rules, and a set of expanded dataset metadata, the set of expanded dataset metadata derived from client input and first dataset metadata, the first dataset metadata derived from the subject dataset, the statement chunking rules indicative of a dimension based chunking for data statements;
expanding a dataset metadata into a set of expanded dataset metadata;
consulting the expanded dataset metadata to perform at least one of determining the at least one chunking scheme, or generating one or more data operations;
based on consulting the expanded dataset metadata, generating the one or more data operations from the data statements, the data operations generated based at least in part on the chunking scheme and on one or more dimensions associated with the subject dataset based on the expanded dataset metadata;
accessing the performance data;
via the performance data, generating first performance estimates associated with the set of data statements;
applying the statement chunking rules to the first performance estimates to determine whether to invoke chunking on the set of data statements, and if so,
applying select statement chunking rules for accessing the set of expanded dataset metadata based on field specific attributes in the expanded dataset metadata to determine chunking parameters for chunking the set of data statements, the select statement chunking rules denoting the first performance estimates based on metrics derived from field specific performance, the select statements derived from the set of data statements;
generating a set of candidate chunking schemes from the chunking parameters;
accessing the performance data to generate second performance estimates for the set of candidate chunking schemes;
selecting the chunking scheme from the set of candidate chunking schemes based on the second performance estimates;
and
executing the data operations over the subject dataset to generate a result set.

14 more elements

A method comprising:

receiving a set of data statements issued by a client, the set of data statements issued by the client to operate over a subject dataset;
analyzing the set of data statements to determine statement attributes associated with the data statements;
expanding dataset metadata into a set of expanded dataset metadata;
accessing a set of client-specific data, the set of client-specific data including performance data, statement chunking rules, and the set of expanded dataset metadata, the set of expanded dataset metadata derived at least in part from client input and dataset metadata, the dataset metadata derived from the subject dataset the statement chunking rules indicative of a dimension based chunking for data statements;
generating a performance predictive model derived from the performance data, the performance data including historical data operations, performance statistics, and historical data operations behavioral characteristics;
applying the performance predictive model to the set of statement attributes to generate performance estimates;
applying the statement chunking rules to the performance estimates to determine whether to invoke chunking on the set of data statements;
applying select statement chunking rules for accessing the set of expanded dataset metadata based on field specific attributes in the expanded dataset metadata to determine chunking parameters for chunking the set of data statements,
the select statement chunking rules denoting the performance estimates based on metrics derived from field specific performance, the select statements derived from the set of data statements,
the chunking parameters including a set of dimensions, a set of measures, a set of relationships, a set of hierarchies, a set of additivity characteristics, a set of cardinality characteristics and a set of structure characteristics;
generating a set of candidate chunking schemes from the chunking parameters;
accessing the performance data to generate performance estimates for the set of candidate chunking schemes;
selecting the chunking scheme from the set of candidate chunking schemes based on the performance estimates;
consulting the expanded dataset metadata to perform at least one of determining the at least one chunking scheme, or generating one or more data operations,
based on consulting the expanded dataset metadata,
generating a set of data operations from the set of data statements, the set of data operations generated based at least in part on the chunking scheme and one or more dimensions associated with the subject dataset based on the expanded dataset metadata;
determining execution directives for the set of data operations, the execution directives indicating how to execute the set of data operations;
executing the set of data operations over the subject dataset to generate results;
and
merging the results from the set of data operations into a result set.

17 more elements

Dependencies are inferred from the claim text.