# US 2019/0294596 A1 · Incremental Feature Development and Workload Capture in Database Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0294596 A1
- Document ID: US20190294596A1
- Status: Published application
- Kind: A1
- Application number: 16/359,452
- Filed: 2019-03-20
- Published: 2019-09-26
- Earliest claimed: 2018-03-22
- Assignees: Snowflake Inc.
- Applicants: Snowflake Inc.
- Inventors: Jiaqi Yan; Qiuye Jin; Shrainik Jain; Stralis Viglas; Allison Lee
- Web page: https://patentgrep.com/patent/US20190294596A1

## CPC

- G06F 16/217: {Database tuning (G06F16/2282 takes precedence; database performance monitoring G06F11/3409)}
- G06F 11/0712: {in a virtual computing platform, e.g. logically partitioned systems}
- G06F 11/0772: {Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers}
- G06F 11/301: {where the computing system is a virtual computing platform, e.g. logically partitioned systems (virtual machines G06F9/45533; logical partitioning of resources G06F9/5077)}
- G06F 11/3072: {where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting}
- G06F 11/3428: {Benchmarking}
- G06F 11/3452: {Performance evaluation by statistical analysis}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3692: {for test results analysis}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F 16/21: Design, administration or maintenance of databases
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/24549: {Run-time optimisation}
- G06F 16/24568: {Data stream processing; Continuous queries}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and devices for incremental feature development and workload capture in database systems are disclosed. A method includes determining a workload having one or more historical client queries to be rerun for testing the feature, wherein the feature comprises procedural logic. The method further includes executing a baseline run of the workload that does not implement the feature and executing a target run of the workload while implementing the feature. The method further includes comparing the baseline run and the target run to identify whether there is a performance regression in the target run. The method further includes, in response to identifying the performance regression, rerunning the target run to identify whether the performance regression still exists.

## Claims

### Claim 1

A system for testing a feature on a database, the system comprising:
means for determining a workload comprising one or more historical client queries to be rerun for testing the feature, wherein the feature comprises procedural logic;
means for executing a baseline run of the workload that does not implement the feature;
means for executing a target run of the workload while implementing the feature;
means for comparing the baseline run and the target run to identify whether there is a performance regression in the target run; and
in response to identifying the performance regression, means for rerunning the target run to identify whether the performance regression still exists.

### Claim 2 (depends on 1)

The system of claim 1, wherein the means for determining the workload is configured to determine based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries, wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested an historical client query;
a query schema;
a table or micro-partition of a database that may be accessed by a query;
a time window for an historical query to have been requested and/or executed;
a query text fragment;
a query execution configuration; or
an execution time of a query.

### Claim 3 (depends on 2)

The system of claim 2, wherein the means for determining the workload is configured to:
deduplicate historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
prune the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
select a subset of the pruned grouping of historical client queries to determine a diverse sample, wherein the selecting comprises applying a Query2Vec algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 4 (depends on 1)

The system of claim 1, wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.

### Claim 5 (depends on 1)

The system of claim 1, wherein the means for rerunning the target run is configured to rerun using separate isolated resources with lower concurrency, and wherein the system further comprises, in response to identifying that the performance regression no longer exists when the target run is rerun, means for flagging the performance regression as a false positive.

### Claim 6 (depends on 5)

The system of claim 5, wherein the means for comparing the baseline run and the target run is further configured to identify one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein the means for rerunning the target run is further configured to identify whether any of the one or more errors still exists; and
wherein the means for flagging the performance regression as a false positive is further configured to flag any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.

### Claim 7 (depends on 6)

The system of claim 6, further comprising means for generating a report comprising data about the target run, wherein the report does not comprise an indication of any performance regression or error that was flagged as a false positive.

### Claim 8 (depends on 1)

The system of claim 1, further comprising means for obfuscating results of the baseline run and the target run such that a user does not have visibility into any database data associated with the one or more historical client queries of the workload.

### Claim 9 (depends on 1)

The system of claim 1, wherein the means for executing the baseline run and the means for executing the target run are configured to execute on a dedicated virtual warehouse of a cloud database service that is dedicated for performance of feature testing runs.

### Claim 10 (depends on 1)

The system of claim 1, further comprising means for determining a version of a client's database data that existed when an historical client query of the one or more historical client queries was run by the client, and wherein the means for executing the baseline run and the means for executing the target run are configured to execute on the version of the client's database data that existed when the historical client query of the one or more historical client queries was run by the client.

### Claim 11

A method for testing a feature on a database, the method comprising:
determining a workload comprising one or more historical client queries to be rerun for testing the feature, wherein the feature comprises procedural logic;
executing a baseline run of the workload that does not implement the feature;
executing a target run of the workload while implementing the feature;
comparing the baseline run and the target run to identify whether there is a performance regression in the target run; and
in response to identifying the performance regression, rerunning the target run to identify whether the performance regression still exists.

### Claim 12 (depends on 11)

The method of claim 11, wherein determining the workload comprises determining based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries, wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested an historical client query;
a query schema;
a table or micro-partition of a database;
a time window for an historical query to have been requested and/or executed;
a query text fragment; or
a query execution configuration.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining the workload comprises:
deduplicating historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
pruning the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
selecting a subset of the pruned grouping of historical client queries to determine a diverse sample, wherein the selecting comprises applying a Query2Vec algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 14 (depends on 11)

The method of claim 11, wherein parameters for the baseline run and parameters for the target run are identical except for a presence of the feature in the target run.

### Claim 15 (depends on 11)

The method of claim 11, further comprising, in response to identifying that the performance regression no longer exists when the target run is rerun, flagging the performance regression as a false positive.

### Claim 16

A processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising:
determining a workload comprising one or more historical client queries to be rerun for testing the feature, wherein the feature comprises procedural logic;
executing a baseline run of the workload that does not implement the feature;
executing a target run of the workload while implementing the feature;
comparing the baseline run and the target run to identify whether there is a performance regression in the target run; and
in response to identifying the performance regression, rerunning the target run to identify whether the performance regression still exists.

### Claim 17 (depends on 16)

The processor of claim 16, wherein determining the workload comprises determining based on a workload selection configuration comprising one or more parameters input by a user to filter out unwanted historical client queries, wherein the one or more parameters comprises one or more of:
a query sample size;
a query initiator such as an account or user that has requested an historical client query;
a query schema;
a table or micro-partition of a database;
a time window for an historical query to have been requested and/or executed;
a query text fragment; or
a query execution configuration.

### Claim 18 (depends on 17)

The processor of claim 17, wherein determining the workload comprises:
deduplicating historical client queries that satisfy the workload selection configuration to determine a grouping of historical client queries;
pruning the grouping of historical client queries to eliminate queries that consume more than a threshold amount of resources to execute; and
selecting a subset of the pruned grouping of historical client queries to determine a diverse sample, wherein the selecting comprises applying a Query2Vec algorithm to map the pruned grouping of historical client queries to a high dimensional vector space.

### Claim 19 (depends on 16)

The processor of claim 16, wherein the instructions further comprise, in response to identifying that the performance regression no longer exists when the target run is rerun, flagging the performance regression as a false positive.

### Claim 20 (depends on 19)

The processor of claim 19, wherein comparing the baseline run and the target run further comprises identifying one or more errors comprising:
a wrong result to one or more historical client queries executed in the target run;
an internal error in the target run; or
a user error in the target run; and
wherein the instructions further comprise identifying whether any of the one or more errors still exists when the target run is rerun; and
wherein the instructions further comprise flagging any of the one or more errors as a false positive in response to identifying that any of the one or more errors no longer exists when the target run is rerun.
