Method of Detecting False Test Alarms Using Test Step Failure Analysis

In a computer code testing environment, a method of identifying potential false test alarms to a developer for a code base under test the method comprising:

• executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
• as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
• identifying one or more characteristics of the system and integration test failures;
• comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
• identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build;
• and
• based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.

A system for identifying potential false test alarms to a developer for a code base under test the system comprising:

• one or more processors;
• and
• one or more computer-readable media, wherein the one or more computer-readable media comprise computer-executable instructions that can be executed by at least one of the one or more processors to cause the following to be performed:
• executing a code build in a test system that includes computing functionality and computing infrastructure that is able to execute the build wherein executing the code build in the test system comprises running a test case, including a plurality of system and integration tests, including a plurality of test steps, on the code build;
• as a result of executing the code build and running the test case, identifying one or more system and integration test failures, including identifying failures of one or more test steps from the test case;
• identifying one or more characteristics of the system and integration test failures;
• comparing the characteristics of the system and integration test failures to characteristics of a test case known to cause false test alarms when run on an executing code build;
• identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build;
• and
• based on the act of identifying that the characteristics of the system and integration test failures match the characteristics of the test case known to cause false test alarms when run on an executing code build, providing information to a developer indicating that the system and integration test failure is potentially a false test alarm.