# US 2021/0064515 A1 · Deep q-network reinforcement learning for testing case selection and prioritization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0064515 A1
- Document ID: US20210064515A1
- Status: Published application
- Kind: A1
- Application number: 16/998,224
- Filed: 2020-08-20
- Published: 2021-03-04
- Earliest claimed: 2019-08-27
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Jianwu Xu; Haifeng Chen; Yuchen Bian
- Web page: https://patentgrep.com/patent/US20210064515A1

## CPC

- G06F 11/3684: {for test design, e.g. generating new test cases}
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F 11/3692: {for test results analysis}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for automated software test design and implementation. The system and method being able to establish an initial pool of test cases for testing computer code; apply the initial pool of test cases to the computer code in a testing environment to generate test results; preprocess the test results into a predetermined format; extract metadata from the test results; generate a training sequence; calculate a reward value for the pool of test cases; input the training sequence and reward value into a reinforcement learning agent; utilizing the value output from the reinforcement learning agent to produce a ranking list; prioritizing the initial pool of test cases and one or more new test cases based on the ranking list; and applying the prioritized initial pool of test cases and one or more new test cases to the computer code in a testing environment to generate test results.

## Claims

### Claim 1

A computer implemented method for automated software test design and implementation, comprising:
establishing an initial pool of test cases for testing computer code;
applying the initial pool of test cases to the computer code in a testing environment to generate test results;
preprocessing the test results into a predetermined format;
extracting metadata from the test results;
generating a training sequence;
calculating a reward value for the pool of test cases;
inputting the training sequence and reward value into a reinforcement learning agent;
utilizing the value output from the reinforcement learning agent to produce a ranking list;
prioritizing the initial pool of test cases and one or more new test cases based on the ranking list; and
applying the prioritized initial pool of test cases and one or more new test cases to the computer code in a testing environment to generate test results.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein the reward value, r, is calculated
using
r
=
{
∑
i
=
1
F

x
i
∈
F
x
i
F
*
P
if


F
,
P
≠
0
0
,
if


F
=
0
1
,
if


P
=
0
,
where F is the total number of failed testing cases, P is the total number of passed testing cases, each xi is a member of a set of failed testing cases, and i is the index.

### Claim 3 (depends on 1)

The method as recited in claim 1, further comprising selecting a subset of the set of available test cases based on the ranking assigned to each test case.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein preprocessing the test results includes converting the test results into a JSON (JavaScript Object Notation) format.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein the extracted metadata includes historical results from each of the test cases and a time duration for each of the test cases.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the training sequence generates training samples from a software testing results dump file and the extracted metadata.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the reinforcement learning agent selects and removes particular testing cases from the pool of test cases to form a revised subset of test cases.

### Claim 8

A non-transitory computer readable storage medium comprising a computer readable program for a computer implemented automated software test design and implementation, wherein the computer readable program when executed on a computer causes the computer to perform the steps of:
establishing an initial pool of test cases for testing computer code;
applying the initial pool of test cases to the computer code in a testing environment to generate test results;
preprocessing the test results into a predetermined format;
extracting metadata from the test results;
generating a training sequence;
calculating a reward value, r, for the pool of test cases;
inputting the training sequence and reward value into a reinforcement learning agent;
utilizing the value output from the reinforcement learning agent to produce a ranking list;
prioritizing the initial pool of test cases and one or more new test cases based on the ranking list; and
applying the prioritized initial pool of test cases and one or more new test cases to the computer code in a testing environment to generate test results.

### Claim 9 (depends on 8)

The computer readable program as recited in claim 8, wherein the reward value is calculated using
r
=
{
∑
i
=
1
F

x
i
∈
F
x
i
F
*
P
if


F
,
P
≠
0
0
,
if


F
=
0
1
,
if


P
=
0
,
where F is the total number of failed testing cases, P is the total number of passed testing cases, each xi is a member of a set of failed testing cases, and i is the index.

### Claim 10 (depends on 8)

The method as recited in claim 8, further comprising selecting a subset of the set of available test cases based on the ranking assigned to each test case.

### Claim 11 (depends on 8)

The computer readable program as recited in claim 8, wherein preprocessing the test results includes converting the test results into a JSON (JavaScript Object Notation) format.

### Claim 12 (depends on 11)

The computer readable program as recited in claim 11, wherein the extracted metadata includes historical results from each of the test cases and a time duration for each of the test cases.

### Claim 13 (depends on 12)

The computer readable program as recited in claim 12, wherein the training sequence generates training samples from a software testing results dump file and the extracted metadata.

### Claim 14 (depends on 13)

The computer readable program as recited in claim 13, wherein the reinforcement learning agent selects and removes particular testing cases from the pool of test cases to form a revised subset of test cases.

### Claim 15

A system for automated software test design and implementation, comprising:
a computer system including:
random access memory configured to store an automated software test design and implementation system;
one or more processor devices and an operating system; and
a database configured to store an initial pool of test cases, wherein the automated software test design and implementation system comprises:
a tester configured to applying the initial pool of test cases to computer code to be tested in a testing environment to generate test results;
a preprocessor configured to preprocess the test results into a predetermined format;
an extractor configured to extract metadata from the test results;
a sequence generator configured to generate a training sequence;
a reward generator configured to calculate a reward value, r, for the pool of test cases;
a learning agent configured to receive the training sequence and reward value and output a vector value;
a selector configured to utilize the vector value output from the reinforcement learning agent to produce a ranking list, prioritize the initial pool of test cases and one or more new test cases based on the ranking list, and apply the prioritized initial pool of test cases and one or more new test cases to the computer code in a testing environment to generate test results.

### Claim 16 (depends on 15)

The system as recited in claim 15, wherein the reward value is calculated
using
r
=
{
∑
i
=
1
F

x
i
∈
F
x
i
F
*
P
if


F
,
P
≠
0
0
,
if


F
=
0
1
,
if


P
=
0
,
where F is the total number of failed testing cases, P is the total number of passed testing cases, each xi is a member of a set of failed testing cases, and i is the index.

### Claim 17 (depends on 15)

The system as recited in claim 15, further comprising selecting a subset of the set of available test cases based on the ranking assigned to each test case.

### Claim 18 (depends on 15)

The system as recited in claim 15, wherein preprocessing the test results includes converting the test results into a JSON (JavaScript Object Notation) format.

### Claim 19 (depends on 15)

The system as recited in claim 15, wherein the extracted metadata includes historical results from each of the test cases and a time duration for each of the test cases.

### Claim 20 (depends on 15)

The system as recited in claim 15, wherein the reinforcement learning agent selects and removes particular testing cases from the pool of test cases to form a revised subset of test cases.
