# US 12,216,628 B2 · Optimized identification of performance regression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,628 B2
- Document ID: US12216628B2
- Status: Granted
- Kind: B2
- Application number: 18/470,706
- Filed: 2023-09-20
- Published: 2024-01-11
- Granted: 2025-02-04
- Earliest claimed: 2018-03-22
- Assignees: Snowflake Inc.
- Applicants: Snowflake Inc.
- Inventors: Allison Lee; Shrainik Jain; Qiuye Jin; Stratis Viglas; Jiaqi Yan
- Web page: https://patentgrep.com/patent/US12216628B2

## CPC

- G06F 16/217: {Database tuning (G06F16/2282 takes precedence; database performance monitoring G06F11/3409)}
- G06F 11/0712: {in a virtual computing platform, e.g. logically partitioned systems}
- G06F 11/0772: {Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers}
- G06F 11/301: {where the computing system is a virtual computing platform, e.g. logically partitioned systems (virtual machines G06F9/45533; logical partitioning of resources G06F9/5077)}
- G06F 11/3072: {where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting}
- G06F 11/3428: {Benchmarking}
- G06F 11/3452: {Performance evaluation by statistical analysis}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3692: {for test results analysis}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F 16/21: Design, administration or maintenance of databases
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/24549: {Run-time optimisation}
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system to identify optimal cloud resources for executing workloads. The system deduplicates historical client queries based on a workload selection configuration to determine a grouping of historical client queries. The system generates a workload based on at least a portion of the grouping of historical client queries. The system repeatedly executes a test run of the workload using resources of a cloud environment to determine whether there is a performance difference in the test run. The system, in response to determining that there is no performance difference, identifies one or more sets of decreased resources of the cloud environment. The system re-executes the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment.

## Claims

### Claim 1

A system comprising:
a processor to:
deduplicate historical client queries based on a workload selection configuration to determine a grouping of historical client queries;
generate a workload based on at least a portion of the grouping of historical client queries;
repeatedly execute a test run of the workload using resources of a cloud environment to determine whether there is a performance difference in the test run;
in response to determining that there is no performance difference, identify one or more sets of decreased resources of the cloud environment; and
re-execute the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment.

### Claim 2 (depends on 1)

The system of claim 1, wherein the re-execute the test run using the one or more decreased resources of the cloud environment to determine whether there is the performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment comprises:
selecting a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.

### Claim 3 (depends on 1)

The system of claim 1, wherein the test run implements a feature that comprises procedural logic.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processor to repeatedly execute the test run of the workload using the resources of the cloud environment to determine whether there is the performance difference in the test run comprises:
compare a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on the workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor to:
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 6 (depends on 1)

The system of claim 1, wherein the test run implements a feature that modifies a function of a database.

### Claim 7 (depends on 4)

The system of claim 4, wherein the test run implements a feature, and wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more sets of decreased resources of the cloud environment have a lower concurrency than the resources of the cloud environment.

### Claim 9 (depends on 4)

The system of claim 4, wherein the processor to identify one or more errors in response to comparing the baseline run and the target run, the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run, and an internal error in the target run; or
a user error in the target run; and
wherein to flag the performance difference as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.

### Claim 10 (depends on 9)

The system of claim 9, wherein the processor is further to generate a report comprising data about the target run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.

### Claim 11 (depends on 4)

The system of claim 4, wherein the processor is further to obfuscate results of the baseline run and the target run such that the user does not have visibility into any database data associated with the workload.

### Claim 12 (depends on 1)

The system of claim 1, wherein the processor to execute the test run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.

### Claim 13 (depends on 11)

The system of claim 11, wherein the processor further to:
determine a version of database data that existed when a historical client query was run.

### Claim 14

A method, the method comprising:
deduplicating historical client queries based on a workload selection configuration to determine a grouping of historical client queries;
generating a workload based on at least a portion of the grouping of historical client queries;
repeatedly executing a test run of the workload using resources of a cloud environment to determine whether there is a performance difference in the test run;
in response to determining that there is no performance difference, identifying one or more sets of decreased resources of the cloud environment; and
re-executing the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment.

### Claim 15 (depends on 14)

The method of claim 14, wherein the re-executing the test run using the one or more decreased resources of the cloud environment to determine whether there is the performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment comprises:
selecting a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.

### Claim 16 (depends on 14)

The method of claim 14, wherein the test run implements a feature that comprises procedural logic.

### Claim 17 (depends on 14)

The method of claim 14, wherein repeatedly executing the test run of the workload using resources of the cloud environment to determine whether there is the performance difference in the test run comprises:
comparing a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on the workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
pruning the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
selecting a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 19 (depends on 14)

The method of claim 14, wherein the test run implements a feature that modifies a function of a database.

### Claim 20 (depends on 17)

The method of claim 17, wherein the test run implements a feature, and wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.

### Claim 21 (depends on 14)

The method of claim 14, wherein the one or more sets of decreased resources of the cloud environment have a lower concurrency than the resources of the cloud environment.

### Claim 22 (depends on 17)

The method of claim 17, further comprising:
identifying one or more errors in response to comparing the baseline run and the target run, wherein the one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run, and an internal error in the target run; or
a user error in the target run; and
wherein to flag the performance difference as a false positive, the processor is further to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.

### Claim 23 (depends on 22)

The method of claim 22, further comprising:
generating a report comprising data about the target run, wherein the report does not comprise an indication of any performance difference or error that was flagged as a false positive.

### Claim 24 (depends on 14)

The method of claim 14, further comprising:
executing the test run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.

### Claim 25

A non-transitory computer readable storage media, programmable to execute instructions that, when executed, cause a processor to:
deduplicate historical client queries based on a workload selection configuration to determine a grouping of historical client queries;
generate a workload based on at least a portion of the grouping of historical client queries;
repeatedly execute a test run of the workload using resources of a cloud environment to determine whether there is a performance difference in the test run;
in response to determining that there is no performance difference, identify one or more sets of decreased resources of the cloud environment; and
re-execute the test run using the one or more sets of decreased resources of the cloud environment to determine whether there is a performance difference in the test run that is attributed to the one or more sets of decreased resources of the cloud environment.

### Claim 26 (depends on 25)

The non-transitory computer readable storage media of claim 25, wherein the instructions, when executed by the processor, further cause the processor to:
select a query of the test run based on an execution rate of the query associated with repeatedly executing the test run of the workload using the resources of the cloud environment.

### Claim 27 (depends on 25)

The non-transitory computer readable storage media of claim 25, wherein test run implements a feature that comprises procedural logic.

### Claim 28 (depends on 25)

The non-transitory computer readable storage media of claim 25, wherein the instructions, when executed by the processor, further cause the processor to:
compare a baseline run of the workload and a target run of the workload to identify whether there is a performance difference in the target run, wherein the baseline run and the target run are compared based on the workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries.

### Claim 29 (depends on 25)

The non-transitory computer readable storage media of claim 25, wherein the instructions, when executed by the processor, further cause the processor to:
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample by applying an algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 30 (depends on 25)

The non-transitory computer readable storage media of claim 25, wherein the instructions, when executed by the processor, further cause the processor to:
execute the target run on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.