# US 11,907,106 B2 · Code integration with time-variant test failure detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,106 B2 - Document ID: US11907106B2 - Status: Granted - Kind: B2 - Application number: 17/561,492 - Filed: 2021-12-23 - Published: 2023-06-29 - Granted: 2024-02-20 - Earliest claimed: 2021-12-23 - Assignees: GM Cruise Holdings LLC - Applicants: GM Cruise Holdings LLC - Inventors: Eben M Freeman; Chau Vu; Sireesha Sura - Web page: https://patentgrep.com/patent/US11907106B2 ## CPC - G06F 11/368: {for test version control, e.g. updating test cases to a new software version} - G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F 11/3692: {for test results analysis} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Software code is tested to determine relative performance of the software code. Successfully executing a test may be used to flag problems in a code repository to be corrected or criteria for accepting code for acceptance to the repository. To further evaluate test results, the test may be run several times on the same code version to determine a test result stability score describing how the test results may vary in different executions, such as a frequency that the code passes the test. Based on the score, additional tests may be run to further refine test result stability score. To evaluate time-variance of testing, when a code version fails a test, a previous version of the code may be run with the same test and the results compared to determine if a new regression was introduced. ## Claims ### Claim 1 A method for verifying or rejecting a test result failure of software code, comprising: identifying a first version of the software code and a test protocol for testing the first version of the software code; running the test protocol on the first version of the software code a plurality of times at a first time; receiving first test results from running the test protocol on the first version of the software code; determining a first test result stability score based on the first test results; identifying that the first test result stability score does not meet a pass threshold of the test protocol; and in response to identifying that the first test result stability score does not meet the pass threshold: identifying a previous version of the software code created before the first time and of which the first version of the software code is a modification; running the test protocol on the previous version of the software code a plurality of times to determine second test results; determining a second test result stability score based on the second test results; and determining that the first version of the software code did not introduce a regression relative to the previous version of the software code based on the first test results, the second test results, the first test result stability score, and the second test result stability score. ### Claim 2 (depends on 1) The method of claim 1, wherein the previous version of the software code successfully met the pass threshold when the test protocol was run on the previous version of the software code at a previous time to the first time. ### Claim 3 (depends on 1) The method of claim 1, wherein the first version of the software code is identified, and the test protocol is run responsive to a request to commit the first version of the software code to a code repository. ### Claim 4 (depends on 3) The method of claim 3, wherein the first version of the software code is automatically accepted to be committed to the code repository after determining the first version of the software code did not introduce a regression. ### Claim 5 (depends on 3) The method of claim 3, wherein the first version of the software code is automatically rejected to be committed to the code repository when it is determined that the first version of the software code did not introduce a regression. ### Claim 6 (depends on 1) The method of claim 1, wherein: the first test result stability score is based on a first variation in subscores in the first test results; the second test result stability score is based on a second variation in subscores in the second test results; and the subscores are compared against a threshold in the test protocol. ### Claim 7 (depends on 1) The method of claim 1, wherein the first and second test result stability scores are based on a frequency that results of the first version of the software code and previous version of the software code respectively meet a pass criteria in running the test protocol the plurality of times. ### Claim 8 (depends on 1) The method of claim 1, wherein the test protocol run on the software code may generate results that vary with time. ### Claim 9 A system comprising: a processor; and a non-transitory computer-readable storage medium containing instructions for execution by the processor for: identifying a first version of software code and a test protocol for testing the first version of the software code; running the test protocol on the first version of the software code a plurality of times at a first time; receiving first test results from running the test protocol on the first version of the software code; determining a first test result stability score based on the first test results; identifying that the first test result stability score does not meet a pass threshold of the test protocol; and in response to identifying that the first test result stability score does not meet the pass threshold: identifying a previous version of the software code created before the first time and of which the first version of the software code is a modification; running the test protocol on the previous version of the software code a plurality of times to determine second test results; determining a second test result stability score based on the second test results; and determining that the first version of the software code did not introduce a regression relative to the previous version of the software code based on the first test results, the second test results, the first test result stability score, and the second test result stability score. ### Claim 10 (depends on 9) The system of claim 9, wherein the previous version of the software code successfully met the pass threshold when the test protocol was run on the previous version of the software code at a previous time to the first time. ### Claim 11 (depends on 9) The system of claim 9, wherein the first version of the software code is identified, and the test protocol is run responsive to a request to commit the first version of the software code to a code repository. ### Claim 12 (depends on 11) The system of claim 11, wherein the first version of the software code is automatically accepted to be committed to the code repository after determining the first version of the software code did not introduce a regression. ### Claim 13 (depends on 11) The system of claim 11, wherein the first version of the software code is automatically rejected to be committed to the code repository when it is determined that the first version of the software code did introduce a regression. ### Claim 14 (depends on 9) The system of claim 9, wherein: the first test result stability score is based on a first variation in subscores in the first test results; the second test result stability score is based on a second variation in subscores in the second test results; and the subscores are compared against a threshold in the test protocol. ### Claim 15 (depends on 9) The system of claim 9, wherein the first and second test result stability scores are based on a frequency that results of the first version of the software code and previous version of the software code respectively meet a pass criteria in running the test protocol the plurality of times. ### Claim 16 (depends on 9) The system of claim 9, wherein the test protocol run on the software code may generate results that vary with time. ### Claim 17 One or more non-transitory computer-readable storage media containing instructions executable by one or more processors for: identifying a first version of software code and a test protocol for testing the first version of the software code; running the test protocol on the first version of the software code a plurality of times at a first time; receiving first test results from running the test protocol on the first version of the software code; determining a first test result stability score based on the first test results; identifying that the first test result stability score does not meet a pass threshold of the test protocol; and in response to identifying that the first test result stability score does not meet the pass threshold: identifying a previous version of software code created before the first time and of which the first version of the software code is a modification; running the test protocol on the previous version of the software code a plurality of times to determine second test results; determining a second test result stability score based on the second test results; and determining that the first version of the software code did not introduce a regression relative to the previous version of the software code based on the first test results, the second test results, the first test result stability score, and the second test result stability score. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable storage media of claim 17, wherein the previous version of the software code successfully met a pass threshold when the test protocol was run on the previous version of the software code at a previous time to the first time. ### Claim 19 (depends on 17) The one or more non-transitory computer-readable storage media of claim 17, wherein the first version of the software code is identified, and the test protocol is run responsive to a request to commit the first version of the software code to a code repository. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable storage media of claim 19, wherein the first version of the software code is automatically accepted to be committed to the code repository after determining the first wade-version of the software code did not introduce a regression.