# US 10,504,037 B1 · Systems and methods for automated document review and quality control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,037 B1
- Document ID: US10504037B1
- Status: Granted
- Kind: B1
- Application number: 15/087,946
- Filed: 2016-03-31
- Granted: 2019-12-10
- Earliest claimed: 2016-03-31
- Assignees: Veritas Technologies LLC
- Applicants: Veritas Technologies LLC
- Inventors: Bashyam Anant; Mat Beer; Houshang Livian; Indira Iyer
- Web page: https://patentgrep.com/patent/US10504037B1

## CPC

- G06N 20/00: Machine learning
- G06F 16/24578: {using ranking}
- G06F 16/93: Document management systems
- G06N 3/09: Supervised learning
- G06Q 10/101: Collaborative creation, e.g. joint development of products or services
- G06Q 50/18: Legal services
- G06N 3/08: Learning methods
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer-implemented method for automated document review and quality control may include (1) dividing a set of documents to be reviewed for relevancy into sets of control documents, training documents, quality-control documents, and review documents, (2) calculating, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, (3) assigning, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers, (4) training a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents, and (5) using the predictive model to evaluate the set of review documents. Various other methods, systems, and computer-readable media are also disclosed.

## Claims

### Claim 1

A computer-implemented method for automated document review and quality control, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
dividing a set of documents to be reviewed for relevancy into:
a set of control documents;
a set of training documents;
a set of quality-control documents; and
a set of review documents;
calculating, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer's ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;
assigning, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;
training a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents; and
using the predictive model to evaluate the set of review documents.

### Claim 2 (depends on 1)

The method of claim 1, wherein dividing the set of documents comprises selecting representative samples of the set of documents for the set of control documents and the set of training documents.

### Claim 3 (depends on 1)

The method of claim 1, wherein the set of quality-control documents comprises documents selected from both the set of control documents and the set of training documents for additional review.

### Claim 4 (depends on 1)

The method of claim 1, wherein the set of reviews performed by the group of reviewers comprises reviews received from at least three reviewers from the group of reviewers for each document in the set of quality-control documents.

### Claim 5 (depends on 4)

The method of claim 4, wherein the effective speed score comprises:
an accuracy score based on an agreement between the three reviewers for each document reviewed by the reviewer; and
a speed score based on an average time taken by the reviewer to review the documents.

### Claim 6 (depends on 1)

The method of claim 1, wherein assigning the set of control documents and the set of training documents to the group of reviewers comprises dividing a quantity of documents among the group of reviewers in proportion to the effective speed score for each reviewer.

### Claim 7 (depends on 1)

The method of claim 1, wherein training the predictive model comprises at least one of:
using machine learning to train the predictive model with the set of training reviews;
determining that the predictive model passes a benchmark for accuracy; and
determining that the predictive model fails the benchmark for accuracy.

### Claim 8 (depends on 7)

The method of claim 7, further comprising, in response to determining that the predictive model fails the benchmark:
adding at least one document from the set of review documents to the set of training documents;
adding a review of the document to the set of training reviews; and
retraining the predictive model using the set of training reviews.

### Claim 9 (depends on 1)

The method of claim 1, wherein using the predictive model to evaluate the set of review documents comprises using the predictive model to calculate a prediction rank for each document in the set of review documents.

### Claim 10 (depends on 9)

The method of claim 9, wherein the prediction rank comprises a likelihood score that the document is classified as relevant.

### Claim 11 (depends on 9)

The method of claim 9, further comprising:
identifying a set of relevant documents based on a predetermined threshold for the prediction rank; and
assigning the set of relevant documents to the group of reviewers based on the effective speed score.

### Claim 12 (depends on 11)

The method of claim 11, wherein assigning the set of relevant documents to the group of reviewers comprises assigning a document with a lower prediction rank to a reviewer with a higher effective speed score.

### Claim 13

A system for automated document review and quality control, the system comprising:
a division module, stored in memory, that divides a set of documents to be reviewed for relevancy into:
a set of control documents;
a set of training documents;
a set of quality-control documents; and
a set of review documents;
a calculation module, stored in memory, that calculates; based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer's ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;
an assignment module, stored in memory, that assigns, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;
a training module, stored in memory, that trains a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents;
an evaluation module; stored in memory, that uses the predictive model to evaluate the set of review documents; and
at least one processor that executes the division module, the calculation module, the assignment module, the training module, and the evaluation module.

### Claim 14 (depends on 13)

The system of claim 13, wherein the division module divides the set of documents by selecting representative samples of the set of documents for the set of control documents and the set of training documents.

### Claim 15 (depends on 13)

The system of claim 13, wherein the set of quality-control documents comprises documents selected from both the set of control documents and the set of training documents for additional review.

### Claim 16 (depends on 13)

The system of claim 13, wherein the set of reviews performed by the group of reviewers comprises reviews received from at least three reviewers from the group of reviewers for each document in the set of quality-control documents.

### Claim 17 (depends on 16)

The system of claim 16, wherein the effective speed score comprises:
an accuracy score based on an agreement between the three reviewers for each document reviewed by the reviewer; and
a speed score based on an average time taken by the reviewer to review the documents.

### Claim 18 (depends on 13)

The system of claim 13, wherein the assignment module assigns the set of control documents and the set of training documents to the group of reviewers by dividing a quantity of documents among the group of reviewers in proportion to the effective speed score for each reviewer.

### Claim 19 (depends on 13)

The system of claim 13, wherein the training module trains the predictive model by at least one of:
using machine learning to train the predictive model with the set of training reviews;
determining that the predictive model passes a benchmark for accuracy; and
determining that the predictive model fails the benchmark for accuracy.

### Claim 20

A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
divide a set of documents to be reviewed for relevancy into:
a set of control documents;
a set of training documents;
a set of quality-control documents; and
a set of review documents;
calculate, based on a set of reviews performed by a group of reviewers on the set of quality-control documents, an effective speed score for each reviewer in the group of reviewers, wherein the effective speed score indicates a reviewer's ability to accurately complete a review in a given time, wherein accuracy is determined by agreement between the reviewer and other reviewers;
assign, based on the effective speed score, the set of control documents and the set of training documents to the group of reviewers;
train a predictive model using a set of training reviews performed by the group of reviewers on both the set of training documents and the set of control documents; and
use the predictive model to evaluate the set of review documents.