# US 2024/0202604 A1 · Machine learning techniques for validating and mutating outputs from predictive systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202604 A1
- Document ID: US20240202604A1
- Status: Published application
- Kind: A1
- Application number: 18/081,242
- Filed: 2022-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Optum, Inc.
- Applicants: Optum, Inc.
- Inventors: Ame Osotsi; Sae Goon Lee; Jason R. Robinson
- Web page: https://patentgrep.com/patent/US20240202604A1

## CPC

- G06N 20/20: Ensemble learning
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various embodiments of the present disclosure provide methods, apparatus, systems, computing devices, computing entities, and/or the like for revising classifier predictions, wherein classification labels are tokenized according to an index of predefined classification labels. An embedding is created for each token using an embedding model. The embeddings are provided to a machine learning model that accepts a sequence of tokens as input and produces a sequence of tokens as output. Tokens are extracted from the output of the machine learning model and a classification label corresponding to each token is retrieved according to the index of predefined classification labels.

## Claims

### Claim 1

A computer-implemented method for processing predictive system output, the computer-implemented method comprising:
receiving, by a computing device and originating from a predictive system, predictive output data, the predictive output data comprising a plurality classification labels;
generating, by the computing device, an input sequence of tokens, the input sequence of tokens comprising a tokenization of the plurality of classification labels based at least in part on an index of predefined classification labels;
generating, by the computing device and using a prediction validation machine learning framework, a mutation of the input sequence of tokens, wherein:
the prediction validation machine learning framework comprises an embedding machine learning model and a mutation machine learning model, wherein:
(i) the embedding machine learning model is configured to (a) generate, for each token of the input sequence of tokens, a token embedding, and (b) concatenate each token embedding into a token embedding matrix,
(ii) the mutation machine learning model is configured to generate an output matrix, comprising an output sequence of tokens, based at least in part on the token embedding matrix,
(iii) the mutation of the input sequence of tokens comprises retrieval of one or more output classification labels associated with the output sequence of tokens based at least in part on the index of predefined classification labels; and
performing, by the computing device, one or more prediction-based actions based at least in part on the mutation of the input sequence of tokens.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein training the mutation machine learning model comprises imparting the mutation machine learning model with a plurality of training sequence of tokens pairs, each training sequence of tokens pair comprising a training input sequence of tokens and a training output sequence of tokens.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the mutation machine learning model comprises a sequence-to-sequence machine learning model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the token embedding matrix comprises a first dimension associated with an embedding size and a second dimension associated with a quantity of tokens associated with the input sequence of tokens.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the output matrix comprises a first dimension associated with a quantity of possible predictions based at least in part on the index of predefined classification labels and a second dimension associated with a quantity of predictions associated with the output sequence of tokens.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the output sequence of tokens is based at least in part on position values in the first dimension including a highest value for each position in the second dimension.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, the one or more output classification labels is based at least in part on integer values in the index of predefined classification labels.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the index of predefined classification labels comprises a universal set of classification labels mapped to integer values.

### Claim 9

An apparatus for processing predictive system output, the apparatus comprising at least one processor and at least one memory including program code, the at least one memory and the program code configured to, with the processor, cause the apparatus to at least:
receive predictive output data from a predictive system, the predictive output data comprising a plurality classification labels;
generate an input sequence of tokens, the input sequence of tokens comprising a tokenization of the plurality of classification labels based at least in part on an index of predefined classification labels;
generate, using a prediction validation machine learning framework, a mutation of the input sequence of tokens, wherein:
the prediction validation machine learning framework comprises an embedding machine learning model and a mutation machine learning model, wherein:
(i) the embedding machine learning model is configured to (a) generate, for each token of the input sequence of tokens, a token embedding, and (b) concatenate each token embedding into a token embedding matrix,
(ii) the mutation machine learning model is configured to generate an output matrix, comprising an output sequence of tokens, based at least in part on the token embedding matrix,
(iii) the mutation of the input sequence of tokens comprises retrieval of one or more output classification labels associated with the output sequence of tokens based at least in part on the index of predefined classification labels; and
perform one or more prediction-based actions based at least in part on the mutation of the input sequence of tokens.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein training the mutation machine learning model comprises imparting the mutation machine learning model with a plurality of training sequence of tokens pairs, each training sequence of tokens pair comprising a training input sequence of tokens and a training output sequence of tokens.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the mutation machine learning model comprises a sequence-to-sequence machine learning model.

### Claim 12 (depends on 9)

The apparatus of claim 9, wherein the token embedding matrix comprises a first dimension associated with an embedding size and a second dimension associated with a quantity of tokens associated with the input sequence of tokens.

### Claim 13 (depends on 9)

The apparatus of claim 9, wherein the output matrix comprises a first dimension associated with a quantity of possible predictions based at least in part on the index of predefined classification labels and a second dimension associated with a quantity of predictions associated with the output sequence of tokens.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the output sequence of tokens is based at least in part on position values in the first dimension including a highest value for each position in the second dimension.

### Claim 15 (depends on 14)

The apparatus of claim 14, the one or more output classification labels is based at least in part on integer values in the index of predefined classification labels.

### Claim 16 (depends on 9)

The apparatus of claim 9, wherein the index of predefined classification labels comprises a universal set of classification labels mapped to integer values.

### Claim 17

A computer program product for processing predictive system output, the computer program product comprising at least one non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions configured to:
receive predictive output data from a predictive system, the predictive output data comprising a plurality classification labels;
generate an input sequence of tokens, the input sequence of tokens comprising a tokenization of the plurality of classification labels based at least in part on an index of predefined classification labels;
generate, using a prediction validation machine learning framework, a mutation of the input sequence of tokens, wherein:
the prediction validation machine learning framework comprises an embedding machine learning model and a mutation machine learning model, wherein:
(i) the embedding machine learning model is configured to (a) generate, for each token of the input sequence of tokens, a token embedding, and (b) concatenate each token embedding into a token embedding matrix,
(ii) the mutation machine learning model is configured to generate an output matrix, comprising an output sequence of tokens, based at least in part on the token embedding matrix,
(iii) the mutation of the input sequence of tokens comprises retrieval of one or more output classification labels associated with the output sequence of tokens based at least in part on the index of predefined classification labels; and
perform one or more prediction-based actions based at least in part on the mutation of the input sequence of tokens.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the output matrix comprises a first dimension associated with a number of possible predictions based at least in part on the index of predefined classification labels and a second dimension associated with a quantity of predictions associated with the output sequence of tokens.

### Claim 19 (depends on 18)

The computer program product of claim 18, wherein the output sequence of tokens is based at least in part on position values in the first dimension including a highest value for each position in the second dimension.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein the one or more output classification labels is based at least in part on integer values in the index of predefined classification labels.
