# US 2021/0342258 A1 · Automated testing of program code under development

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342258 A1
- Document ID: US20210342258A1
- Status: Published application
- Kind: A1
- Application number: 16/864,429
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Micro Focus LLC
- Applicants: Micro Focus LLC
- Inventors: Sigal Ishay; Ilan Shufer; Sharon Lin
- Web page: https://patentgrep.com/patent/US20210342258A1

## CPC

- G06F 11/3692: {for test results analysis}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3698: Environments for analysis, debugging or testing of software
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Automated test failures that result from automated testing of program code under development are windows to include just the automated test failures occurring for a first time and that are due to automated test code defects or program code defects. The automated test failures that remain after winnowing are clustered into automated test failure clusters that each individually corresponding to a different automated test code defect or a different program code defect. The automated test failure clusters are window to include just the automated test failure clusters that each individually correspond to a different program code defect. The automated test failure clusters that remain after winnowing are output.

## Claims

### Claim 1

A method comprising:
winnowing a plurality of automated test failures that result from automated testing of program code under development to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;
clustering the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect;
winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failure clusters that each individually correspond to a different automated test code defect are removed; and
outputting the automated test failure clusters that remain after winnowing.

### Claim 2 (depends on 1)

The method of claim 1, wherein outputting the automated test failure clusters comprises:
outputting a ratio of a number of the automated test failure clusters to a number of manually identified program code defects within the program code, as a ratio of a number of automated testing-identified program code defects within the program code to the number of manually identified program code defects.

### Claim 3 (depends on 2)

The method of claim 2, wherein outputting the automated test failure clusters further comprises:
outputting a graph of the ratio of the automated testing-identified program code defects to the number of manually identified program code defects over time.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
modifying the automated testing of the program code under development to increase the ratio of the ratio of the automated testing-identified program code defects to the number of manually identified program code defects.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
modifying the automated testing of the program code under development to improve the automated testing.

### Claim 6 (depends on 1)

The method of claim 1, wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:
winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to an environment in which the automated testing of the program code under development was conducted, the issues being failures within hardware of the environment, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

### Claim 7 (depends on 1)

The method of claim 1, wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:
winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to instability of the automated testing of the program code under development, including an inability of the automated test code in handling race conditions, resulting in multiple operations being performed simultaneously instead of in sequence, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

### Claim 8 (depends on 1)

The method of claim 1, wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:
winnowing the automated test failures to include the automated test failures identified as regression automated test failures.

### Claim 9 (depends on 1)

The method of claim 1, wherein winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect comprises:
for each automated test failure cluster, determining whether modification of the automated test code that was executed to perform the automated testing of the program code under development resolved the automated test failures of the automated test failure cluster; and
excluding from the automated failure test clusters each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code, in that each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code is due to an automated test code defect and not due to a program code defect.

### Claim 10 (depends on 9)

The method of claim 9, wherein, for each automated test failure cluster, determining whether modification of the automated test code resolved the automated test failures of the automated test failure cluster comprises:
determining when the automated test failures of the automated test failure cluster last occurred during the automated testing of the program code;
determining when the automated test failures of the automated test failure cluster first no longer occurred during the automated testing of the program code;
determining whether the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred; and
responsive to determining that the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred, determining that modification of the automated test code resolved the automated test failures of the automated test failure cluster.

### Claim 11

A non-transitory computer-readable data storage medium storing program code executable by a processor to:
determine a number of program code defects identified within program code under development by automated testing of the program code; and
output the number of program code defects identified within the program code by the automated testing relative to a number of program code defects manually identified within the program code under development, by outputting a ratio of the number of program code defects identified within the program code by the automated testing to the number of program code defects manually identified within the program code.

### Claim 12

(canceled)

### Claim 13 (depends on 11)

The non-transitory computer-readable data storage medium of claim 11, wherein the processor is to output the number of program code defects identified within the program code by the automated testing relative to the number of program code defects manually identified within the program code by further:
outputting a graph of the ratio of the number of program code defects identified within the program code by the automated testing to the number of program code defects manually identified within the program code over time.

### Claim 14 (depends on 11)

The non-transitory computer-readable data storage medium of claim 11, wherein the processor is to determine the number of program code defects identified within the program code under development by the automated testing by:
winnowing a plurality of automated test failures that result from the automated testing to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;
clustering the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect; and
winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failures that each individually correspond to a different automated test code defect are removed.

### Claim 15 (depends on 14)

The non-transitory computer-readable data storage medium of claim 14, wherein winnowing the automated test failures to include just the automated test failures occurring for the first time and that are due to the automated test code defects and the program code defects comprises:
winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to an environment in which the automated testing of the program code under development was conducted, the issues being failures within hardware of the environment, such that the automated test failures that result from the issues are not indicative of defects within the program code under development; and
winnowing the automated test failures to exclude the automated test failures that result from issues pertaining to instability of the automated testing of the program code under development, including an inability of the automated test code in handling race conditions, resulting in multiple operations being performed simultaneously instead of in sequence, such that the automated test failures that result from the issues are not indicative of defects within the program code under development.

### Claim 16 (depends on 14)

The non-transitory computer-readable data storage medium of claim 14, wherein winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect comprises:
for each automated test failure cluster, determining whether modification of automated test code that was executed to perform the automated testing of the program code under development resolved the automated test failures of the automated test failure cluster; and
excluding from the automated test failure clusters each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code, in that each automated test failure cluster of which the automated test failures were resolved by modification of the automated test code is due to an automated test code defect and not due to a program code defect.

### Claim 17

A system comprising:
a processor; and
a memory storing program code executable by the processor, the program code comprising:
automated testing analysis program code to determine a number of program code defects identified within program code under development by automated testing of the program code, and output the number of program code defects identified within the program code by the automated testing relative to a number of program code defects manually identified within the program code under development, by outputting a ratio of the number of program code defects identified within the program code by the automated testing to the number of program code defects manually identified within the program code.

### Claim 18 (depends on 17)

The system of claim 17, wherein the program code further comprises:
automated testing program code to perform the automated testing on the program code under development; and
development program code to modify the program code under development.

### Claim 19

(canceled)

### Claim 20 (depends on 17)

The system of claim 17, wherein the automated testing analysis program code is to determine the number of program code defects identified within the program code under development by the automated testing by:
winnowing a plurality of automated test failures that result from the automated testing to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;
clustering the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect; and
winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failures that each individually correspond to a different automated test code defect are removed.

### Claim 21 (depends on 16)

The non-transitory computer-readable data storage medium of claim 16, wherein, for each automated test failure cluster, determining whether modification of the automated test code resolved the automated test failures of the automated test failure cluster comprises:
determining when the automated test failures of the automated test failure cluster last occurred during the automated testing of the program code;
determining when the automated test failures of the automated test failure cluster first no longer occurred during the automated testing of the program code;
determining whether the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred; and
responsive to determining that the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred, determining that modification of the automated test code resolved the automated test failures of the automated test failure cluster.

### Claim 22 (depends on 20)

The system of claim 20, wherein the automated testing analysis program code is to determine the number of program code defects identified within the program code under development by the automated testing by:
winnowing a plurality of automated test failures that result from the automated testing to include just the automated test failures occurring for a first time and that are due to automated test code defects and program code defects, such that the automated test failures resulting from causes other than the automated test code defects and the program code defects are removed, the automated test code defects being defects within automated test code executed to perform the automated testing of the program code under development, and the program code defects being defects within the program code under development;
clustering the automated test failures that remain after winnowing into a plurality of automated test failure clusters that each individually correspond to a different automated test code defect or a different program code defect; and
winnowing the automated test failure clusters to include just the automated test failure clusters that each individually correspond to a different program code defect, such that the automated test failures that each individually correspond to a different automated test code defect are removed,
and wherein, for each automated test failure cluster, determining whether modification of the automated test code resolved the automated test failures of the automated test failure cluster comprises:
determining when the automated test failures of the automated test failure cluster last occurred during the automated testing of the program code;
determining when the automated test failures of the automated test failure cluster first no longer occurred during the automated testing of the program code;
determining whether the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred; and
responsive to determining that the automated test code was modified between when the automated test failures of the automated test failure cluster last occurred and when the automated test failures of the automated test failure cluster first no longer occurred, determining that modification of the automated test code resolved the automated test failures of the automated test failure cluster.
