# US 2024/0202057 A1 · Methods and systems for determining stopping point

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202057 A1
- Document ID: US20240202057A1
- Status: Published application
- Kind: A1
- Application number: 18/595,261
- Filed: 2024-03-04
- Published: 2024-06-20
- Earliest claimed: 2019-10-23
- Assignees: Relativity Oda LLC
- Applicants: Relativity Oda LLC
- Inventors: Jesse Allan Winkler; Elise Tropiano; Robert Jenson Price; Brandon Gauthier; Theo Van Wijk; Patricia Ann Gleason
- Web page: https://patentgrep.com/patent/US20240202057A1

## CPC

- G06N 3/091: Active learning
- G06N 20/00: Machine learning
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06Q 50/18: Legal services
- G06F 11/3438: {monitoring of user actions (tracking the activity of the user H04L67/535)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method includes based on a calculated first estimated error rate, second estimated error rate, first uncertain rank count, second uncertain rank count, and target error rate, displaying a stopping point indication. A computing system includes a processor; and a memory storing instructions that, when executed, cause the computing system to: based on a calculated first estimated error rate, second estimated error rate, first uncertain rank count, second uncertain rank count, and target error rate, display a stopping point indication.

## Claims

### Claim 1

A computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising:
calculating a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate,
displaying, in a display of a computing device, an indication that the stopping point has been reached.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes receiving a coverage review indication from a user.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of documents.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of document groups.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the target error rate is a configurable constant.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein calculating the first uncertain rank count and the second uncertain rank count includes comparing uncertain rank counts across a configurable number of previous builds.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein displaying, in the display of the computing device, the indication that the stopping point has been reached includes
generating a message indicating that the stopping point has been reached; and
transmitting the message via one or both of (i) a push message, and (ii) an email message.

### Claim 8

A computing system for determining a stopping point of a machine learning process, comprising:
one or more processors; and
a memory storing instructions that, when executed, cause the computing system to:
calculate a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate,
display, in a display of a computing device, an indication that the stopping point has been reached.

### Claim 9 (depends on 8)

The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to:
receive a coverage review indication from a user.

### Claim 10 (depends on 8)

The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to:
determine whether a user has coded a minimum number of documents.

### Claim 11 (depends on 8)

The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to:
determine whether a user has coded a minimum number of document groups.

### Claim 12 (depends on 8)

The computing system of claim 8, wherein the target error rate is a configurable constant.

### Claim 13 (depends on 8)

The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to:
compare uncertain rank counts across a configurable number of previous builds.

### Claim 14 (depends on 8)

The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to:
generate a message indicating that the stopping point has been reached; and
transmit the message via one or both of (i) a push message, and (ii) an email message.