Ranking and optimizing automated test scripts
AssigneeMicrosoft Corporation
InventorsKevin Rocco Beto, Che Yan Timothy Wong, Ricky Kurniawan
Life of this patent
Estimating the term…
Abstract
Technologies are described herein for ranking and optimizing test scripts utilized in the automated testing of software products. A score is calculated for each test script from various metrics collected from executions of the test script. Metrics utilized in calculating the score for a test script may include those indicating the propensity of the script to result in “false failures” and/or those indicating the effectiveness of the script for finding product bugs. The test scripts are then ranked by their score, and this ranking is used in determining the frequency of execution of the test script in future testing.
Claims11
A computer-implemented method for ranking and optimizing automated test scripts, the method comprising:
- executing, on one or more computer systems, a plurality of test scripts, wherein the test scripts are members of a collection;
- recording, at a server computer, a test result and execution data in a repository for each of the plurality of test scripts executed;
- calculating, by the server computer, a score for each of the plurality of test scripts by multiplying a number of executions of the test script within a first period of time by a sum of a plurality of terms comprising a ratio of the number of executions of the test script resulting in a test result of failure not due to product bug to the number of executions of the test script within the first period of time, a ratio of the number of executions of the test script resulting in the test result of failure not due to product bug to the number of executions of the test script within a second period of time, and the number of executions of the test script resulting in a test result of failure due to product bug or failure not due to product bug within the first period of time based upon the test results and execution data recorded in the repository for the test script;
- ranking, by the server computer, the plurality of test scripts according to the score calculated for each;
- determining the members of the collection based upon the rank of each of the plurality of test scripts;
- and
- repeating the method over one or more iterations of testing.
A computer-implemented method for ranking and optimizing automated test scripts, the method comprising:
- executing, on one or more computer systems, a plurality of test scripts, wherein the test scripts are members of a collection;
- recording, at a server computer, a test result and execution data in a repository for each of the plurality of test scripts executed;
- calculating, by the server computer, a score for each of the plurality of test scripts by adding a plurality of terms comprising an average of an execution time of the test script over a period of time, a number of executions of the test script resulting in a test result of failure due to product bug within the period of time, a ratio of the number of executions of the test script resulting in a test result of failure not due to product bug to the number of executions of the test script within the period of time, and a number of unique lines of code tested by the test script based upon the test results and execution data recorded in the repository for the test script, wherein each of the plurality of terms is multiplied by a weighting factor;
- ranking, by the server computer, the plurality of test scripts according to the score calculated for each;
- determining the members of the collection based upon the rank of each of the plurality of test scripts;
- and
- repeating the method over one or more iterations of testing.
A computer storage medium that is not a signal, having computer executable instructions stored thereon that, when executed by a computer, cause the computer to:
- collect in a repository a test result and execution data for a plurality of executions of each of a plurality of test scripts;
- calculate a score for each of the plurality of test scripts by adding a plurality of terms comprising an average of an execution time of the test script over a period of time, a number of executions of the test script resulting in a test result of failure due to product bug within the period of time, a ratio of the number of executions of the test script resulting in a test result of failure not due to product bug to the number of executions of the test script within the period of time, and a number of unique lines of code tested by the test script based upon the test results and execution data;
- rank the plurality of test scripts according to the score;
- present to a test reviewer a list of the plurality test scripts sorted according to the rank;
- receive from the test reviewer a change in frequency of execution for a subset of test scripts in the list;
- and
- repeat the collecting, calculating, ranking, presenting, and receiving over one or more iterations of testing.
Dependencies are inferred from the claim text.