# US 2017/0286276 A1 · Method of Detecting False Test Alarms Using Test Step Failure Analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286276 A1
- Document ID: US20170286276A1
- Status: Published application
- Kind: A1
- Application number: 15/625,636
- Filed: 2017-06-16
- Published: 2017-10-05
- Earliest claimed: 2015-09-04
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Kim Sebastian HERZIG; Nachiappan NAGAPPAN
- Web page: https://patentgrep.com/patent/US20170286276A1

## CPC

- G06F 11/3672: {Test management}
- G06F 11/3692: {for test results analysis}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Identifying false test alarms to a developer. A code build is executed in a test system that includes computing functionality and computing infrastructure that is able to execute the build. Executing the code build includes running a plurality of system and integration tests on the code build. As a result of executing the code build, a system and integration test failure is identified. One or more characteristics of the system and integration test failure are identified. The characteristics of the system and integration test failure are compared to characteristics of a set of historical previous known false test alarms. False test alarms are failures caused by a factor other than a factor for which a test is being run. Based on the act of comparing, information is provided to a developer with respect to if the system and integration test failure is potentially a false test alarm.

## Claims

### Claim 1

A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
a test system that includes computing functionality and computing infrastructure that is able to execute a code build, including running a plurality of system and integration tests on the code build to identify system and integration test failures and to identify one or more characteristics of the system and integration test failures;
a data store, wherein the data store comprises a set of association rules used for identifying potential false test alarms, wherein each association rule in the association rules identifies characteristics of a test case, including system and integration test steps, that results in a false test alarm when system and integration test steps in test cases having the identified characteristics are executed by the test system; and
a comparator coupled to the test system and the data store, wherein the comparator is configured to receive identification of characteristics of system and integration test failures from the test system and to compare the system and integration test failure characteristics to the association rules in the data store to identify potential false test alarms; and
wherein the comparator is configured to generate a notification to a user identifying a test failure as a potential false test alarm when a potential false test alarm is identified.

### Claim 2 (depends on 1)

The system of claim 1, wherein the comparator is configured to compare identified failing test steps in a test case to the association rules.

### Claim 3 (depends on 1)

The system of claim 1, wherein the comparator is configured to provide a statistical analysis of comparisons made by the comparator.

### Claim 4 (depends on 1)

The system of claim 1, wherein the comparator is configured to provide information based on a confidence and a support value.

### Claim 5 (depends on 1)

The system of claim 1, wherein the comparator is configured to provide a ranking of test failures based on the comparisons to the false test alarms.

### Claim 6 (depends on 5)

The system of claim 5, wherein the ranking is based on the statistical significance of the comparison.

### Claim 7 (depends on 1)

The system of claim 1, wherein the comparator is configured to identify potential false test alarms to a user only when a statistical threshold is met.

### Claim 8 (depends on 1)

The system of claim 1, further comprising a learning engine coupled to the data store, wherein the learning engine is configured to identify new association rules and to add the new association rules to the data store.

### Claim 9

In a computer code testing environment, a method of identifying potential false test alarms to a developer for a code base under test the method comprising:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
identifying one or more characteristics of the system and integration test failures;
comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build; and
based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.

### Claim 10 (depends on 9)

The method of claim 9, wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

### Claim 11 (depends on 9)

The method of claim 9, wherein providing information comprises providing a statistical analysis.

### Claim 12 (depends on 9)

The method of claim 9, wherein providing information comprises providing information based on a confidence and a support value.

### Claim 13 (depends on 9)

The method of claim 9, further comprising providing a ranking of test failures based on the comparisons to the false test alarms

### Claim 14 (depends on 13)

The method of claim 13, wherein the ranking is based on the statistical significance of the comparison.

### Claim 15 (depends on 9)

The method of claim 9, wherein providing information is only performed when a statistical threshold is met.

### Claim 16 (depends on 9)

The method of claim 9, further comprising receiving input regarding additional false test alarms, and as a result adding new false test alarms to the set of historical previous known false test alarms to allow new rules to be learned for evaluating subsequent system and integration test failures.

### Claim 17

A system for identifying potential false test alarms to a developer for a code base under test the system comprising:
one or more processors; and
one or more computer-readable media, wherein the one or more computer-readable media comprise computer-executable instructions that can be executed by at least one of the one or more processors to cause the following to be performed:
executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
identifying one or more characteristics of the system and integration test failures;
comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build; and
based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.

### Claim 18 (depends on 17)

The system of claim 17, wherein the compared characteristics are identified failing test steps in a test case such that comparing comprises attempting match failed test steps of a test case for the system and integration test failure to failed test steps of a test case for a historical previous known false test alarm.

### Claim 19 (depends on 17)

The system of claim 17, wherein providing information comprises providing a statistical analysis.

### Claim 20 (depends on 17)

The system of claim 17, wherein providing information comprises providing information based on a confidence and a support value.
