Continuous cloud-scale query optimization and processing
AssigneeMicrosoft Technology Licensing, LLC
InventorsNicolas Bruno, Jingren Zhou
Life of this patent
Estimating the term…
Abstract
Runtime statistics from the actual performance of operations on a set of data are collected and utilized to dynamically modify the execution plan for processing a set of data. The operations performed are modified to include statistics collection operations, the statistics being tailored to the specific operations being quantified. Optimization policy defines how often optimization is attempted and how much more efficient an execution plan should be to justify transitioning from the current one. Optimization is based on the collected runtime statistics but also takes into account already materialized intermediate data to gain further optimization by avoiding reprocessing.
Claims20
One or more computer-readable storage devices comprising computer-executable instructions for optimizing a performance of a data processing job, the computer-executable instructions directed to steps comprising:
- selecting a computer-executable program that includes a first set of operations to carry out the data processing job;
- augmenting at least some operations, of the first set of operations, in the computer-executable program to comprise computer-executable instructions for collecting execution statistics quantifying execution of the augmented operations, the execution statistics being indicative of at least the runtime performance of the first set of operations;
- receiving at least some of the execution statistics prior to the data processing job having been completed;
- selecting, based on the received execution statistics, a second set of operations that are semantically equivalent to the first set of operations, the semantically equivalent second set of operations different but functionally equivalent to the first set of operations and configured to provide a same result as the first set of operations;
- and
- proceeding with the data processing job with the second set of operations.
One or more computer-readable storage devices comprising computer-executable instructions for transitioning from execution of a first set of operations to execution of a second set of operations, the second set of operations being semantically equivalent to the first set of operations, the transitioning occurring after commencing, but prior to completing, a data processing job, the computer-executable instructions directed to steps comprising:
- selecting one or more operations from among the second set of operations, the selected one or more operations included in a computer-executable program;
- searching the first set of operations for a semantically equivalent one or more operations;
- comparing the selected one or more operations from the second set of operations to a found semantically equivalent one or more operations from the first set of operations, if the searching finds the semantically equivalent one or more operations, the found semantically equivalent one or more operations from the first set of operations different but functionally equivalent to the selected one or more operations from the second set of operations and configured to provide a same result as the selected one or more operations from the second set of operations;
- replacing the semantically equivalent one or more operations from the first set of operations with the selected one or more operations from the second set of operations if the comparing finds differences;
- and
- adding the selected one or more operations from the second set of operations if the searching does not find semantically equivalent one or more operations in the first set of operations.
A system for implementing a data processing job, the system comprising:
- an optimizer, at least partially implemented by at least one or more processing units, configured to perform steps comprising:
- selecting a computer-executable program that includes a first set of operations to carry out the data processing job;
- augmenting at least some operations, of the first set of operations, in the computer-executable program to comprise computer-executable instructions for collecting execution statistics quantifying execution of the augmented operations, the execution statistics being indicative of at least the runtime performance of the first set of operations;
- receiving at least some of the execution statistics prior to the data processing job having been completed;
- selecting, based on the received execution statistics, a second set of operations that are semantically equivalent to the first set of operations, the semantically equivalent second set of operations different but functionally equivalent to the first set of operations and configured to provide a same result as the first set of operations;
- and
- proceeding with the data processing job with the second set of operations;
- and
- a scheduler configured to perform steps comprising:
- selecting one or more operations from among the second set of operations;
- searching the first set of operations for a semantically equivalent one or more operations;
- comparing the selected one or more operations from the second set of operations to a found semantically equivalent one or more operations from the first set of operations, if the searching finds the semantically equivalent one or more operations;
- replacing the semantically equivalent one or more operations from the first set of operations with the selected one or more operations from the second set of operations if the comparing finds differences;
- and
- adding the selected one or more operations from the second set of operations if the searching does not find semantically equivalent one or more operations in the first set of operations.
Dependencies are inferred from the claim text.