# US 2021/0263932 A1 · System and method for machine learning for system deployments without performance regressions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0263932 A1
- Document ID: US20210263932A1
- Status: Published application
- Kind: A1
- Application number: 16/840,205
- Filed: 2020-04-03
- Published: 2021-08-26
- Earliest claimed: 2020-02-21
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Irene Rogan Shaffer; Remmelt Herbert Lieve Ammerlaan; Gilbert Antonius; Marc T. Friedman; Abhishek Roy; Lucas Rosenblatt; Vijay Kumar Ramani; Shi Qiao; Alekh Jindal; Peter Orenberg; H M Sajjad Hossain; Soundararajan Srinivasan; Hiren Shantilal Patel; Markus Weimer
- Web page: https://patentgrep.com/patent/US20210263932A1

## CPC

- G06F 16/2453: Query optimisation
- G06F 16/24542: {Plan optimisation}
- G06F 11/3409: {for performance assessment}
- G06F 11/3466: {Performance evaluation by tracing or monitoring}
- G06F 16/24549: {Run-time optimisation}
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods of machine learning for system deployments without performance regressions are performed by systems and devices. A performance safeguard system is used to design pre-production experiments for determining the production readiness of learned models based on a pre-production budget by leveraging big data processing infrastructure and deploying a large set of learned or optimized models for its query optimizer. A pipeline for learning and training differentiates the impact of query plans with and without the learned or optimized models, selects plan differences that are likely to lead to most dramatic performance difference, runs a constrained set of pre-production experiments to empirically observe the runtime performance, and finally picks the models that are expected to lead to consistently improved performance for deployment. The performance safeguard system enables safe deployment not just for learned or optimized models but also for additional of other ML-for-Systems features.

## Claims

### Claim 1

A method performed by a computing system, the method comprising:
determining, for each of a plurality of query sets in a plurality of query jobs, a query set feature representation, based on a first intermediate representation of features corresponding to each of a plurality of default query plan graphs and a second intermediate representation of features corresponding to each of a plurality of respective optimized query plan graphs;
executing a first subset of the query sets;
generating execution data of each query set of the first subset, the first subset selected based on a difference, determined by a difference model, between a default value of a performance metric and an optimized value of the performance metric associated with each of the query sets with respective ones of the query set feature representations; and
deploying, to a query optimizer in a database host, optimized query models corresponding to a second subset of the query sets that is selected from the first subset based on a retrained version of the difference model that is retrained according to the execution data.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating:
the plurality of default query plan graphs respectively based on the query sets of the plurality of query sets and corresponding default query models of a plurality of query models, and
first meta-information respectively associated with each of the default query plan graphs; and
generating:
the plurality of optimized query plan graphs respectively based on query sets of the plurality of query sets and corresponding optimized query models of the plurality of query models, and
second meta-information respectively associated with each of the optimized query plan graphs.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating the first intermediate representation of features in each node corresponding to each of the default query plan graphs; and
generating a second intermediate representation of features in each node corresponding to each of the optimized query plan graphs.

### Claim 4 (depends on 3)

The method of claim 3, wherein said generating the first intermediate representation of features is performed based on the respective first meta-information; and
wherein said generating a second intermediate representation of features is performed based on the respective second meta-information.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
performing at least one additional iteration of said executing and said generating, prior to said deploying, that each generates additional execution data, each additional iteration being performed on a respective iterative additional subset ultimately selected from the first subset that is based on a respective iterative additional retrained version of the difference model that is retrained according to the additional execution data of an associated iteration; and
wherein the retrained version of the difference model of said deploying is retrained according to a most recent version of the execution data from a latest iteration of said executing.

### Claim 6 (depends on 1)

The method of claim 1, wherein the difference model is based on a graph convolution associated with the first intermediate representations, default query plan graphs, second intermediate representations, and optimized query plan graphs; or
wherein said deploying further comprises:
querying a database by the database host utilizing the deployed optimized query models of the query optimizer.

### Claim 7

A system that comprises:
a processing system comprising one or more processors; and
a memory configured to store program instructions that, when executed, configure the processing system to:
generate a first subset of query jobs, from a plurality of query jobs having a plurality of query sets respectively, based on one or more subset criteria;
generate an updated difference model by training a difference model based on execution data received from executed query sets corresponding to the first subset, the difference model configured to determine a difference between a default performance metric and an optimized performance metric corresponding to a query set of a query job based on query set feature representations associated therewith;
generate, for each graph pair of graph pairs of the first subset having a first performance metric corresponding to a default query plan graph and a second performance metric corresponding to an optimized query plan graph of respective graphs of the graph pairs, and according to the updated difference model, an updated first performance metric and an updated second performance metric based on respective query set feature representations associated therewith;
generate a second subset of query jobs by selecting one or more query jobs of the first subset according to a comparison of a determined initial difference score to an updated difference score between the updated first performance metric and the updated second performance metric; and
provide, to a query optimizer, optimized query models associated with at least one query job of the second subset as a deployment model set.

### Claim 8 (depends on 7)

The system of claim 7, wherein the processing system is further configured to:
receive the graph pairs, each comprising a respective default query plan graph corresponding to a default query model and a respective optimized query plan graph corresponding to an optimized query model with both being based on one of the query sets.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processing system is further configured to:
receive query set feature representations, each of which respectively correspond to the graph pairs and are based on intermediate representations of features in nodes of the respective default query plan graph or the respective optimized query plan graph.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processing system is further configured to:
generate, for each graph pair and according to the difference model, the first performance metric and the second performance metric, based on the associated one of the query set feature representations.

### Claim 11 (depends on 9)

The system of claim 9, wherein the processing system is further configured to:
generate a first importance score for a first node of a first default query plan graph;
generate a second importance score for a second node of a first optimized query plan graph that has a same physical operator as the first node;
determine one or more factors that are indicative of a variation value between the first importance score and the second importance score based on respective intermediate representations of the first default query plan graph and of the first optimized query plan graph according to attention; and
provide at least one of the variation value, the one or more factors, or information associated with the one or more factors in an output report.

### Claim 12 (depends on 7)

The system of claim 7, wherein the one or more subset criteria comprise:
a comparison of a threshold value to a determined difference score between the first performance metric and the second performance metric, or
a subset model that incorporates constraints of a pre-production environment utilized in generating the execution data.

### Claim 13 (depends on 7)

The system of claim 7, wherein the processing system is further configured to:
prior to said provide:
generate a further updated difference model by training the updated difference model based on further execution data received from executed query sets of the plurality of query sets corresponding to the second subset;
generate, for graph pair of graph pairs of the second subset, and according to the further updated difference model, a further updated first performance metric and a further updated second performance metric based on respective query set feature representations associated therewith; and
generate a third subset of query jobs by selecting one or more query jobs of the second subset according to a comparison of a further determined initial difference score to a further updated difference score between the further updated first performance metric and the further updated second performance metric; and
wherein said provide comprises to provide optimized query models associated with at least one query job of the third subset as the deployment model set.

### Claim 14

A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing system, perform a method comprising:
generating a first subset of query jobs, from a plurality of query jobs having a plurality of query sets respectively, based on one or more subset criteria;
generating an updated difference model by training a difference model based on execution data received from executed query sets corresponding to the first subset, the difference model configured to determine a difference between a default performance metric and an optimized performance metric corresponding to a query set of a query job based on query set feature representations associated therewith;
generating, for each graph pair of graph pairs of the first subset having a first performance metric corresponding to a default query plan graph and a second performance metric corresponding to an optimized query plan graph of respective graphs of the graph pairs of the first subset, and according to the updated difference model, an updated first performance metric and an updated second performance metric based on respective query set feature representations associated therewith;
generating a second subset of query jobs by selecting one or more query jobs of the first subset according to a comparison of a determined initial difference score to an updated difference score between the updated first performance metric and the updated second performance metric; and
providing, to a query optimizer, optimized query models associated with at least one query job of the second subset as a deployment model set.

### Claim 15 (depends on 14)

The computer-readable storage medium of claim 14, wherein the method includes:
receiving the graph pairs, each comprising a respective default query plan graph corresponding to a default query model and a respective optimized query plan graph corresponding to an optimized query model with both being based on one of the query sets.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the method includes:
receiving query set feature representations, each of which respectively correspond to graph pairs and are based on intermediate representations of features in nodes of the respective default query plan graph or the respective optimized query plan graph.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the method includes:
generating, for each graph pair and according to the difference model, the first performance metric and the second performance metric, based on the associated one of the query set feature representations.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein the method includes:
generating a first importance score for a first node of a first default query plan graph;
generating a second importance score for a second node of a first optimized query plan graph that has a same physical operator as the first node;
determining one or more factors that are indicative of a variation value between the first importance score and the second importance score based on respective intermediate representations of the first default query plan graph and of the first optimized query plan graph according to attention; and
providing at least one of the variation value, the one or more factors, or information associated with the one or more factors in an output report.

### Claim 19 (depends on 14)

The computer-readable storage medium of claim 14, wherein the one or more subset criteria comprise:
a comparison of a threshold value to a determined difference score between the first performance metric and the second performance metric, or
a subset model that incorporates constraints of a pre-production environment utilized in generating the execution data.

### Claim 20 (depends on 14)

The computer-readable storage medium of claim 14, wherein the method includes:
prior to said providing:
generating a further updated difference model by training the updated difference model based on further execution data received from executed query sets corresponding to the second subset;
generating, for each graph pair of graph pairs of the second subset, and according to the further updated difference model, a further updated first performance metric and a further updated second performance metric based on respective query set feature representations associated therewith; and
generating a third subset of query jobs by selecting one or more query jobs of the second subset according to a comparison of a further determined initial difference score to a further updated difference score between the further updated first performance metric and the further updated second performance metric; and
wherein said providing comprises providing optimized query models associated with at least one query job of the third subset as the deployment model set.
