# US 2021/0279577 A1 · Testing of Computing Processes Using Artificial Intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279577 A1
- Document ID: US20210279577A1
- Status: Published application
- Kind: A1
- Application number: 17/192,651
- Filed: 2021-03-04
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Seva Development, LLC
- Applicants: Seva Development, LLC
- Inventors: Peter Gardiner West; Daniel Moroni Hair; Jeffrey Douglas Handy; Michael Ray Kimball; Daniel James Ricks
- Web page: https://patentgrep.com/patent/US20210279577A1

## CPC

- G06N 3/08: Learning methods
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Technology is described for generating tests to execute against a process. The method can include identifying a data store of input operation requests for the process, and the input operation requests are recorded for requests received to operate functionality of the process. Another operation can be training a deep neural network model using the input operation requests to enable the deep neural network model to generate output series based in part on the input operation requests. Test series can be generated using the deep neural network model. The test series are executable to activate functionality of the process in order to test portions of the process. A further operation may be executing the test series on the process in order to test functionality of the process.

## Claims

### Claim 1

A method for generating tests to execute against a process, comprising:
identifying a data store of input operation requests for the process, wherein the input operation requests are recorded as requests are received to operate functionality of the process;
training a deep neural network model using the input operation requests to enable the deep neural network model to generate test series based in part on the input operation requests;
generating one or more test series using the deep neural network model, wherein the test series are executable to activate functionality of the process and test portions of the process; and
executing the one or more test series on the process in order to test the functionality of the process.

### Claim 2 (depends on 1)

The method as in claim 1, wherein the input operation requests include time series data representing at least one of: input operation requests for a user interface, input operation requests captured graphically for a screen area, or API requests received from another process.

### Claim 3 (depends on 2)

The method as in claim 2, further comprising capturing time series data that includes data fields that are at least one of: a time stamp, session data, a user interface component being activated, a function type being applied, data to be entered, or an area of an interface being activated.

### Claim 4 (depends on 1)

The method as in claim 1, wherein the input operation requests are test input operation requests that test a defined portion of the process' functionality.

### Claim 5 (depends on 1)

The method as in claim 1 wherein the input operation requests are user interactions that have a series of events including events selected from at least one of: clicking a button, selecting data in a drop-down list box, selecting data in a grid, gaining focus on a control, navigating in a control, navigating to a control, typing in a defined field, or dragging an element in a defined way.

### Claim 6 (depends on 1)

The method as in claim 1, further comprising training the deep neural network model that is a transformer-based machine learning model or an attention based deep neural network model.

### Claim 7 (depends on 1)

The method as in claim 1, further comprising training a machine learning model that is at least one of: a GPT model, a GPT-2 model, a bidirectional encoder representations from transformers model (BERT), a conditional transformer language model (CTRL), a lite BERT model (ALBERT), a universal language model fine-tuning (ULMFiT), a transformer architecture, embeddings from language models (ELMo), recurrent neural nets (RNNs), or convolutional neural nets (CNNs).

### Claim 8 (depends on 1)

The method as in claim 1, further comprising processing a test series with a machine learning model using classification to classify whether the test series is executable on the process.

### Claim 9

A system to automatically generate tests to execute on a process, comprising:
a data store of input operation requests for the process, which have been recorded as at least one user utilizes functionality of the process;
a deep neural network model that is trained using the input operation requests;
a test generator to generate one or more test series using the deep neural network model, wherein the one or more test series are executable to activate functionality of the process and to test portions of the process; and
a test validator to validate the one or more test series with a machine learning model used as a classifier to classify whether a test series is executable on the process; and
a test executor to execute validated test series to test functionality of the process.

### Claim 10 (depends on 9)

The system as in claim 9, further comprising:
determining an execution probability for a test series; and
validating the test series with an execution probability that is above a test execution threshold value.

### Claim 11 (depends on 9)

The system as in claim 9, further comprising executing a test on the process to determine validity based on whether the test executes.

### Claim 12 (depends on 9)

The system as in claim 9, wherein the input operation requests are user test series that test a portion of the process' functionality as recorded from a user during a testing session.

### Claim 13 (depends on 9)

The system as in claim 9, further comprising capturing input operation requests that are time series data which includes data fields that are at least one of: a time stamp, session data, a user interface component being activated, a function type being applied, data to be entered, or an area of an interface being activated.

### Claim 14 (depends on 9)

The system as in claim 9, further comprising recording input operation requests in a user interaction session that represent user interaction events with graphical controls or command line controls of a user interface of the process.

### Claim 15 (depends on 9)

The system as in claim 9, further comprising training a deep neural network model that is a transformer-based machine learning model or an attention based deep neural network model.

### Claim 16 (depends on 9)

The system as in claim 9, further comprising training a machine learning model that is at least one of: a GPT model, a GPT-2 model, a bidirectional encoder representations from transformers model (BERT), a conditional transformer language model (CTRL), a lite BERT model (ALBERT), a universal language model fine-tuning (ULMFiT), a transformer architecture, embeddings from language models (ELMo), recurrent neural nets (RNNs), or convolutional neural nets (CNNs).

### Claim 17 (depends on 9)

The system as in claim 9, further comprising:
recording backend operations initiated by the process as sent to backend servers providing services to the process; and
processing the backend operations using a transformer based artificial intelligence model to determine that the backend operations is valid behavior for the process.

### Claim 18 (depends on 17)

The system as in claim 17, wherein the backend operations are at least one of: API calls, server log entries, data store updates, DOM (document object model) updates, POM (page object model) updates, remote function calls, or HTTP requests.

### Claim 19

A non-transitory computer-readable medium comprising computer-executable instructions which implement automatic generation of tests to execute on a process, comprising:
identifying a data store of input operation requests for the process, which have been recorded as requests are received to operate functionality of the process;
training a deep neural network model using the input operation requests to enable the deep neural network model to generate output series based in part on the input operation requests;
generating one or more test series using the deep neural network model, wherein the test series are executable to activate functionality of the process in order to test portions of the process;
executing the one or more test series on the process in order to test functionality of the process;
receiving test output from the process in response to the one or more test series;
processing test output with a machine learning model used as classifier to determine that the test output represents valid behavior of the process; and
reporting when the test output has a set of invalid behavior.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
receiving test output from a front end of the process in response to execution of a test series;
processing test output with a machine learning model used as classifier to determine whether the test output represents valid behavior of the process; and
reporting whether the test output was valid.

### Claim 21 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
receiving test output from a backend of the process in response to execution of a test series;
processing test output with a machine learning model used as classifier to determine whether the test output represents valid behavior of the process; and
reporting whether the test output was valid.

### Claim 22 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
receiving input operation requests for a test for the process;
receiving test output that includes output events resulting from a test for the process;
processing the input operation requests and output events with a machine learning model used as classifier to determine whether the input operation requests and output events represent valid behavior for inputs to the process; and
reporting whether the input operation requests and the test output are valid.

### Claim 23 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
tracking API calls from the process to backend servers in communication with the process; and
processing the API calls with a machine learning model used as classifier to determine that the API calls represent valid behavior of the process.

### Claim 24 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
tracking HTTP requests or server log writes from the process to backend servers in communication with the process; and
processing the HTTP requests or server log writes with a machine learning model used as classifier to determine that the API calls represent valid behavior of the process.

### Claim 25 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
determining that a test failed; and
comparing updates in source code to developer or tester actions to determine that changes are intentional and are not defects.

### Claim 26 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising analyzing visual images from the process to determine whether a test has passed.

### Claim 27 (depends on 26)

The non-transitory computer-readable medium as in claim 26, further comprising analyzing at least one of: text, user interface controls, or images in a visual image of the process.

### Claim 28 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
summarizing user stories or product requirements for a process using a deep neural network model to generate summarized product functionality;
converting visual output of the process to text to form a visual summary of the process; and
comparing the summarized product functionality with the visual summary to determine whether a test failure was intentional.

### Claim 29 (depends on 19)

The non-transitory computer-readable medium as in claim 19, further comprising:
tracking of text values or numerical values across a plurality of electronic pages or a plurality of data stores to identify data relationships to be maintained; and
verifying whether the data relationships were maintained using a test series executed by a test runner.

### Claim 30 (depends on 19)

The non-transitory computer-readable medium as in claim 19, comprising:
identifying a failure of a test;
checking whether a failure of a test was due to an intentional change;
adapting the test when the change is identified as intentional change;
retrying the test; and
marking the test that failed in a retry as the test that failed due to an intentional change and the test was not able to be modified to fix the test.
