US 9,870,314 B1Granted

Update testing by build introspection

AssigneeRed Hat, Inc.

InventorsJay Vyas, Huamin Chen

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present disclosure provides new and innovative methods and systems for ordering tests in an application update environment. For example, an update to an application may be received. A file name of a file affected by the update may be detected. A correlation matrix, which associates file names and the test names with correlations based on test metrics, may be queried with the file name to return a correlation between the file name and a test. A test plan may be built based on the correlation between the detected file name and the test, and may include an ordered list based on correlations from the correlation matrix. The test plan may then be run. A test failure and test metric of a test may be detected and the correlation matrix may be updated with a correlation between the file name and the failed test based on the test metric.

Claims20

Dependent claims

A method comprising:

storing, in a memory, a plurality of files, the plurality of files including a first file and a second file that are used by an application, wherein each file in the plurality of files has a respective file name;
storing, in the memory, a plurality of tests, the plurality of tests including a first test and a second test that are used to test the application, wherein each test in the plurality of tests has a respective test name;
storing, in the memory, a plurality of test results, the plurality of test results including a first test result, which includes a first file name of the first file, which was updated by the application after a first update was applied to the application, a first test name of the first test, and a first test metric based on the first test;
storing, in the memory, a correlation matrix, which associates the first file name and the first test name with a first correlation between the first file and the first test based on the first test metric;
receiving a second update to the application, wherein the second update includes a first test profile including a first plurality of test names designated as test failure candidates;
detecting the first test profile;
detecting a second file name of the second file, which is updated by the application after the second update is applied to the application;
detecting, by querying the correlation matrix with the second file name, a second correlation between the second file and the second test;
building a first test plan based on at least the second correlation and the first test profile, wherein the first test plan includes an ordered list of a second plurality of test names including the second test name and the first plurality of test names, and wherein the second test name and the first plurality of test names are respectively ordered based on at least a plurality of correlations from the correlation matrix including the second correlation;
running the first test plan based on the ordered list;
detecting a first failure of the second test;
generating a second test result, which includes the second file name, the second test name, and a second test metric based on the second test;
and
updating, in the correlation matrix, the second correlation between the second file and the second test, based on the second test metric.

11 more elements

A system comprising:

a memory, wherein the memory stores:
a plurality of files, the plurality of files including a first file and a second file that are used by an application, wherein each file in the plurality of files has a respective file name,
a plurality of tests, the plurality of tests including a first test and a second test that are used to test the application, wherein each test in the plurality of tests has a respective test name,
a plurality of test results, the plurality of test results including a first test result, which includes a first file name of the first file, which was updated by the application after a first update was applied to the application, a first test name of the first test, and a first test metric based on the first test, and
a correlation matrix, which associates the first file name and the first test name with a first correlation between the first file and the first test based on the first test metric;
and
a processor, wherein the processor:
receives a second update to the application, wherein the second update includes a first test profile including a first plurality of test names designated as test failure candidates;
detects the first test profile;
detects a second file name of the second file, which is updated by the application after the second update is applied to the application;
detects, by querying the correlation matrix with the second file name, a second correlation between the second file and the second test;
builds a first test plan based on at least the second correlation and the first test profile, wherein the first test plan includes an ordered list of a second plurality of test names including the second test name and the first plurality of test names, and wherein the second test name and the first plurality of test names are respectively ordered based on at least a plurality of correlations from the correlation matrix including the second correlation;
runs the first test plan based on the ordered list;
detects a first failure of the second test;
generates a second test result, which includes the second file name, the second test name, and a second test metric based on the second test;
and
updates, in the correlation matrix, the second correlation between the second file and the second test, based on the second test metric.

14 more elements

A non-transitory computer readable medium storing instructions,

which when executed, cause a build server to:
store, a plurality of files, the plurality of files including a first file and a second file that are used by an application, wherein each file in the plurality of files has a respective file name;
store a plurality of tests, the plurality of tests including a first test and a second test that are used to test the application, wherein each test in the plurality of tests has a respective test name;
store a plurality of test results, the plurality of test results including a first test result, which includes a first file name of the first file, which was updated by the application after a first update was applied to the application, a first test name of the first test, and a first test metric based on the first test;
store a correlation matrix, which associates the first file name and the first test name with a first correlation between the first file and the first test based on the first test metric;
receive a second update to the application, wherein the second update includes a first test profile including a first plurality of test names designated as test failure candidates;
detect the first test profile;
detect a second file name of the second file, which is updated by the application after the second update is applied to the application;
detect, by querying the correlation matrix with the second file name, a second correlation between the second file and the second test;
build a first test plan based on at least the second correlation and the first test profile, wherein the first test plan includes an ordered list of a second plurality of test names including the second test name and the first plurality of test names, and wherein the second test name and the first plurality of test names are respectively ordered based on at least a plurality of correlations from the correlation matrix including the second correlation;
run the first test plan based on the ordered list;
detect a first failure of the second test;
generate a second test result, which includes the second file name, the second test name, and a second test metric based on the second test;
and
update, in the correlation matrix, the second correlation between the second file and the second test, based on the second test metric.

12 more elements

Dependencies are inferred from the claim text.