# US 2024/0202058 A1 · Methods and systems for determining stopping point

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202058 A1
- Document ID: US20240202058A1
- Status: Published application
- Kind: A1
- Application number: 18/595,290
- Filed: 2024-03-04
- Published: 2024-06-20
- Earliest claimed: 2019-10-23
- Assignees: Relativity Oda LLC
- Applicants: Relativity Oda LLC
- Inventors: Jesse Allan Winkler; Elise Tropiano; Robert Jenson Price; Brandon Gauthier; Theo Van Wijk; Patricia Ann Gleason
- Web page: https://patentgrep.com/patent/US20240202058A1

## CPC

- G06N 3/091: Active learning
- G06N 20/00: Machine learning
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06Q 50/18: Legal services
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method, computing system, and non-transitory computer-readable medium are disclosed for configuring a machine learning-assisted review process. The method includes receiving user-defined parameters, retrieving a set of documents based on these parameters, and displaying the documents for user review and coding. Coding decisions are associated with the documents and used to modify training parameters for the machine learning process, which includes employing various neural network models such as recurrent, convolutional, and deep learning neural networks. The system and medium further involve creating, storing, and adjusting machine learning models based on coding decisions. The process aims to enhance document review efficiency by adapting machine learning models to user feedback, ultimately displaying progress and indicating when a review process has reached a predetermined stopping point.

## Claims

### Claim 1

A computer-implemented method for configuring a machine learning-assisted review process in a computing system, comprising:
receiving, via one or more processors, user-defined parameters corresponding to training parameters for the machine learning-assisted review process;
retrieving, via one or more processors, a set of documents from a communication corpus based on the user-defined parameters;
displaying, via one or more processors, the set of documents in a browser-based interface thereby enabling a user to review and code the documents;
receiving, via one or more processors, a plurality of coding decisions from the user via an input device;
associating, via one or more processors, the coding decisions with the documents;
transmitting, via one or more processors, the coding decisions and document identifiers to a machine learning module;
modifying, via one or more processors, training parameters for the machine learning-assisted review process based on the coding decisions;
storing the coding decisions in the communication corpus;
displaying an indication of the machine learning-assisted review process progress in the browser-based interface; and
displaying an indication that the machine learning-assisted review process has reached a stopping point based on predetermined criteria.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the machine learning module includes instructions for creating, retrieving, and storing machine learning models, and wherein the machine learning models include at least one of recurrent neural networks, convolutional neural networks, and deep learning neural networks.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the machine learning module further includes instructions for serializing and deserializing the machine learning models, and wherein the training of the machine learning model involves adjusting weights of the model based on the coding decisions and document identifiers.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the machine learning module is configured to employ a regression neural network for training, and wherein the training includes normalization of input data by mean centering and employing a mean squared error loss function.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the machine learning module further includes callbacks for regenerating document rankings after incremental training, and wherein the regenerated document rankings are transmitted to a web server for display to the user.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the machine learning module is configured to train the machine learning model using a Bayesian model, and wherein the training includes dividing data sets into training, validation, and testing subsets.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the machine learning model includes an artificial neural network having an input layer, one or more hidden layers, and an output layer, and wherein each layer includes an arbitrary number of neurons configured to process input parameters and generate a prediction.

### Claim 8

A computing system for configuring a machine learning-assisted review process, comprising:
one or more processors;
a memory having stored thereon instructions that, when executed by the one or more processors, cause the computing system to:
receive user-defined parameters corresponding to training parameters for the machine learning-assisted review process;
retrieve a set of documents from a communication corpus based on the user-defined parameters;
display the set of documents in a browser-based interface thereby enabling a user to review and code the documents;
receive a plurality of coding decisions from the user via an input device;
associate the coding decisions with the documents;
transmit the coding decisions and document identifiers to a machine learning module;
modify training parameters for the machine learning-assisted review process based on the coding decisions;
store the coding decisions in the communication corpus;
display an indication of the machine learning-assisted review process progress in the browser-based interface; and
display an indication that the machine learning-assisted review process has reached a stopping point based on predetermined criteria.

### Claim 9 (depends on 8)

The computing system of claim 8, the memory having stored thereon instructions that, when executed by the one or more processors, cause the computing system to:
create, retrieve and store machine learning models, and wherein the machine learning models include at least one of recurrent neural networks, convolutional neural networks, or deep learning neural networks.

### Claim 10 (depends on 8)

The computing system of claim 8, the memory having stored thereon instructions that, when executed by the one or more processors, cause the computing system to:
serialize and deserialize the machine learning models, and wherein the training of the machine learning model involves adjusting weights of the model based on the coding decisions and document identifiers.

### Claim 11 (depends on 8)

The computing system of claim 8, the memory having stored thereon instructions that, when executed by the one or more processors, cause the computing system to:
use a regression neural network for training, wherein the training includes normalization of input data by mean centering and employing a mean squared error loss function.

### Claim 12 (depends on 8)

The computing system of claim 8, wherein the machine learning module further includes callbacks for regenerating document rankings after incremental training, and wherein the regenerated document rankings are transmitted to a web server for display to the user.

### Claim 13 (depends on 8)

The computing system of claim 8, wherein the machine learning module is configured to train the machine learning model using a Bayesian model, and wherein the training includes dividing data sets into training, validation, and testing subsets.

### Claim 14 (depends on 8)

The computing system of claim 8, wherein the machine learning model includes an artificial neural network having an input layer, one or more hidden layers, and an output layer, and wherein each layer includes an arbitrary number of neurons configured to process input parameters and generate a prediction.

### Claim 15

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause a computer to:
receive user-defined parameters corresponding to training parameters for the machine learning-assisted review process;
retrieve a set of documents from a communication corpus based on the user-defined parameters;
display the set of documents in a browser-based interface thereby enabling a user to review and code the documents;
receive a plurality of coding decisions from the user via an input device;
associate the coding decisions with the documents;
transmit the coding decisions and document identifiers to a machine learning module;
modify training parameters for the machine learning-assisted review process based on the coding decisions;
store the coding decisions in the communication corpus;
display an indication of the machine learning-assisted review process progress in the browser-based interface; and
display an indication that the machine learning-assisted review process has reached a stopping point based on predetermined criteria.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, having stored thereon instructions that, when executed by the one or more processors, cause a computer to:
create, retrieve and store machine learning models, and wherein the machine learning models include at least one of recurrent neural networks, convolutional neural networks, or deep learning neural networks.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, having stored thereon instructions that, when executed by the one or more processors, cause a computer to:
serialize and deserialize the machine learning models, and wherein the training of the machine learning model involves adjusting weights of the model based on the coding decisions and document identifiers.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, having stored thereon instructions that, when executed by the one or more processors, cause a computer to:
use a regression neural network for training, wherein the training includes normalization of input data by mean centering and employing a mean squared error loss function.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the machine learning module further includes callbacks for regenerating document rankings after incremental training, and wherein the regenerated document rankings are transmitted to a web server for display to the user.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the machine learning module is configured to train the machine learning model using a Bayesian model, and wherein the training includes dividing data sets into training, validation, and testing subsets.