# US 12,008,598 B2 · Adaptive optimization of a content item using continuously trained machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,598 B2
- Document ID: US12008598B2
- Status: Granted
- Kind: B2
- Application number: 17/889,296
- Filed: 2022-08-16
- Published: 2022-12-08
- Granted: 2024-06-11
- Earliest claimed: 2020-01-03
- Assignees: Claritas, Inc.
- Applicants: Claritas, LLC
- Inventors: Yuri Khidekel; Mikhail Zaleshin; Vladimir Bashmakov
- Web page: https://patentgrep.com/patent/US12008598B2

## CPC

- G06Q 30/0254: {based on statistics}
- G06N 20/20: Ensemble learning
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A processor receives requests for content items and identifies a first subset of machine learning (ML) models that satisfy a reliability criterion and a second subset of ML models that fail to satisfy the reliability criterion, wherein each ML model is associated with a respective content template and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item generated based on the respective associated content template. The processing logic assigns each request to either a first group or a second group based on a ratio of a number of ML models in the first subset to a number of ML models in the second subset. For each request in the first group, the processor generates a content item based on a content template associated with the first subset.

## Claims

### Claim 1

A system comprising:
a memory; and
a processing device, operatively coupled with the memory, the processing device configured to perform operations comprising:
receiving a plurality of requests for content items, wherein each of the plurality of requests is associated with a respective target comprising a respective set of characteristics;
identifying a first subset of a plurality of trained machine learning models that satisfy a reliability criterion and a second subset of the plurality of trained machine learning models that fail to satisfy the reliability criterion, wherein each trained machine learning model of the plurality of trained machine learning models is associated with a respective content template of a plurality of content templates and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item generated based on the respective content template associated with the respective trained machine learning model;
for each request of the plurality of requests, assigning the request to a first group or a second group based on a ratio of a number of trained machine learning models in the first subset to a number of trained machine learning models in the second subset; and
for each request in the first group, generating a content item based on a content template associated with one of the first subset of the plurality of trained machine learning models.

### Claim 2 (depends on 1)

The system of claim 1, wherein generating the content item based on the content template associated with one of the first subset of the plurality of trained machine learning models comprises:
inputting the respective set of characteristics associated with the request into each of the first subset of the plurality of trained machine learning models; and
selecting the content template based on outputs of the first subset of the plurality of trained machine learning models.

### Claim 3 (depends on 2)

The system of claim 2, wherein selecting the content template comprises:
determining, based on the outputs of the first subset of the plurality of trained machine learning models, the content template associated with an output indicating a highest probability of the respective target performing the target action.

### Claim 4 (depends on 1)

The system of claim 1, wherein the operations further comprise:
for each request in the second group, generating a second content item based on a second content template associated with one of the second subset of the plurality of trained machine learning models.

### Claim 5 (depends on 4)

The system of claim 4, wherein generating the second content item based on the second content template associated with one of the second subset of the plurality of trained machine learning models comprises:
determining one of one or more content templates associated with the second subset of the plurality of trained machine learning models; and
generating the second content item in accordance with the determined content template.

### Claim 6 (depends on 1)

The system of claim 1, wherein the operations further comprise:
receiving, for each generated content item for each request in the second group, a respective indication as to whether the associated respective target performed a target action responsive to presentation of the respective content item to the respective target;
training the second subset of the plurality of trained machine learning models, wherein each trained machine learning model of the second subset of the plurality of trained machine learning models is trained for a respective content template of the plurality of content templates using a) respective sets of characteristics of targets associated with requests for which the respective content template was selected, b) respective content items generated for the respective requests and c) respective indications associated with the respective content items;
determining, for each trained machine learning model in the second subset, whether the trained machine learning model satisfies the reliability criterion; and
moving those trained machine learning models that satisfy the reliability criterion from the second subset of the plurality of trained machine learning models to the first subset of the plurality of trained machine learning models.

### Claim 7 (depends on 1)

The system of claim 1, wherein generating the content item based on the content template comprises:
determining a plurality of components of the content template, each component associated with one or more elements;
for each component of the content template having a plurality of elements, performing the following comprising:
identifying a first subset of a second plurality of trained machine learning models that satisfy another reliability criterion and a second subset of the second plurality of trained machine learning models that fail to satisfy the another reliability criterion, wherein each trained machine learning model of the second plurality of trained machine learning models is associated with a respective element of a respective component of the content template and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item including the respective element associated with the respective trained machine learning model of the second plurality of trained machine learning models;
assigning the request to a third group or a fourth group;
for each request in the third group, performing the following comprising:
inputting the respective set of characteristics associated with the request into each of the first subset of the second plurality of trained machine learning models; and
selecting an element to be included in the content item for the respective component of the content template, based on outputs of the first subset of the second plurality of trained machine learning models; and
for each request in the fourth group, selecting an element to be included in the content item for the respective component of the content template, based on an element associated with one of the second subset of the second plurality of trained machine learning models;
for each component of the content template having one element, selecting the one element to be included in the content item for the respective component of the content template; and
generating a content item comprising the selected elements associated with the plurality of components of the content template.

### Claim 8 (depends on 7)

The system of claim 7, wherein a trained machine learning model of the second plurality of trained machine learning models corresponds to a machine learning model associated with an element associated with a component of the content template that is trained using training data comprising training inputs and training outputs,
the training inputs including a plurality set of characteristics associated with targets, and
the training outputs including a plurality of indications as to whether corresponding target actions were performed by the targets responsive to being presented with content items comprising the respective element.

### Claim 9 (depends on 7)

The system of claim 7, wherein assigning the request to the third group or the fourth group comprises:
determining a second ratio of a number of trained machine learning models in the first subset of the second plurality of trained machine learning models to a number of trained machine learning models the second subset of the second plurality of trained machine learning models;
determining a number of requests to assign to the third group and another number of requests to assign to the fourth group based on the second ratio; and
assigning the request to the third group or the fourth group based on the number of requests to assign to the third group and the another number of requests to assign to the fourth group.

### Claim 10 (depends on 7)

The system of claim 7, wherein the operations further comprise:
receiving, for each generated content item for each request in the fourth group, a respective indication as to whether the associated respective target performed a target action responsive to presentation of the respective content item to the respective target;
training the second subset of the second plurality of trained machine learning models, wherein each trained machine learning model of the second subset of the second plurality of trained machine learning models is trained for a respective element of a component of the content template using a) respective sets of characteristics of targets associated with requests for which the respective element was selected, b) respective content items including the associated respective element and c) respective indications associated with the respective content items;
determining, for each trained machine learning model in the second subset of the second plurality of trained machine learning models, whether the trained machine learning model satisfies the reliability criterion; and
moving those trained machine learning models that satisfy the another reliability criterion from the second subset of the second plurality of trained machine learning models to the first subset of the second plurality of trained machine learning models.

### Claim 11 (depends on 10)

The system of claim 10, wherein the another reliability criterion is associated with at least one of a mean or a standard deviation of an area under receiver operating characteristic (ROC) curve determined for a respective trained machine learning model of the second plurality of trained machine learning models using one or more sets of validation data, the validation data comprising validation input data and validation output data, wherein the validation input data includes another set of characteristics associated with another target and the validation output data includes an indication of whether or not another target action was performed.

### Claim 12 (depends on 1)

The system of claim 1, wherein the reliability criterion is associated with at least one of a mean or a standard deviation of an area under receiver operating characteristic (ROC) curve determined for a respective trained machine learning model of the plurality of trained machine learning models using one or more sets of validation data, the validation data comprising validation input data and validation output data, wherein the validation input data includes another set of characteristics associated with another target and the validation output data includes an indication of whether or not another target action was performed.

### Claim 13 (depends on 1)

The system of claim 1, wherein assigning the request to the first group or the second group comprises:
determining the ratio of the number of trained machine learning models in the first group to the number of trained machine learning models the second group;
determining a number of requests to assign to the first group and another number of requests to assign to the second group based on the ratio; and
assigning the request to the first group or the second group based on the number of requests to assign to the first group and the another number of requests to assign to the second group.

### Claim 14 (depends on 1)

The system of claim 1, wherein a trained machine learning model of the plurality of trained machine learning models corresponds to a machine learning model that is associated with the content template and that is trained using training data comprising training inputs and training outputs,
the training inputs including a plurality set of characteristics associated with targets, and
the training outputs including a plurality of indications as to whether corresponding target actions were performed by the targets responsive to being presented with content items based on the content template.

### Claim 15

A method comprising:
receiving a plurality of requests for content items, wherein each of the plurality of requests is associated with a respective target comprising a respective set of characteristics;
identifying a first subset of a plurality of trained machine learning models that satisfy a reliability criterion and a second subset of the plurality of trained machine learning models that fail to satisfy the reliability criterion, wherein each trained machine learning model of the plurality of trained machine learning models is associated with a respective content template of a plurality of content templates and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item generated based on the respective content template associated with the respective trained machine learning model;
for each request of the plurality of requests, assigning the request to a first group or a second group based on a ratio of a number of trained machine learning models in the first subset to a number of trained machine learning models in the second subset; and
for each request in the first group, generating a content item based on a content template associated with one of the first subset of the plurality of trained machine learning models.

### Claim 16 (depends on 15)

The method of claim 15, wherein generating the content item based on the content template associated with one of the first subset of the plurality of trained machine learning models comprises:
inputting the respective set of characteristics associated with the request into each of the first subset of the plurality of trained machine learning models;
selecting the content template based on outputs of the first subset of the plurality of trained machine learning models, wherein the content template is associated with an output indicating a highest probability of the respective target performing the target action; and
generating the content item based on the content template by performing the following comprising:
determining a plurality of components of the content template, each component associated with one or more elements;
for each component of the content template having a plurality of elements, performing the following comprising:
identifying a first subset of a second plurality of trained machine learning models that satisfy another reliability criterion and a second subset of the second plurality of trained machine learning models that fail to satisfy the another reliability criterion, wherein each trained machine learning model of the second plurality of trained machine learning models is associated with a respective element of a respective component of the content template and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item including the respective element associated with the respective trained machine learning model of the second plurality of trained machine learning models;
assigning the request to a third group or a fourth group;
for each request in the third group, performing the following comprising:
inputting the respective set of characteristics associated with the request into each of the first subset of the second plurality of trained machine learning models; and
selecting an element to be included in the content item for the respective component of the content template, based on outputs of the first subset of the second plurality of trained machine learning models; and
for each request in the fourth group, selecting an element to be included in the content item for the respective component of the content template, based on an element associated with one of the second subset of the second plurality of trained machine learning models;
for each component of the content template having one element, selecting the one element to be included in the content item for the respective component of the content template; and
generating a content item comprising the selected elements associated with the plurality of components of the content template.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
for each request in the second group, generating a second content item based on a second content template associated with one of the second subset of the plurality of trained machine learning models.

### Claim 18 (depends on 15)

The method of claim 15, further comprising:
receiving, for each generated content item for each request in the second group, a respective indication as to whether the associated respective target performed a target action responsive to presentation of the respective content item to the respective target;
training the second subset of the plurality of trained machine learning models, wherein each trained machine learning model of the second subset of the plurality of trained machine learning models is trained for a respective content template of the plurality of content templates using a) respective sets of characteristics of targets associated with requests for which the respective content template was selected, b) respective content items generated for the respective requests and c) respective indications associated with the respective content items;
determining, for each trained machine learning model in the second subset, whether the trained machine learning model satisfies the reliability criterion; and
moving those trained machine learning models that satisfy the reliability criterion from the second subset of the plurality of trained machine learning models to the first subset of the plurality of trained machine learning models.

### Claim 19

A non-transitory computer readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:
receiving a plurality of requests for content items, wherein each of the plurality of requests is associated with a respective target comprising a respective set of characteristics;
identifying a first subset of a plurality of trained machine learning models that satisfy a reliability criterion and a second subset of the plurality of trained machine learning models that fail to satisfy the reliability criterion, wherein each trained machine learning model of the plurality of trained machine learning models is associated with a respective content template of a plurality of content templates and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item generated based on the respective content template associated with the respective trained machine learning model;
for each request of the plurality of requests, assigning the request to a first group or a second group based on a ratio of a number of trained machine learning models in the first subset to a number of trained machine learning models in the second subset; and
for each request in the first group, generating a content item based on a content template associated with one of the first subset of the plurality of trained machine learning models.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein generating the content item based on the content template associated with one of the first subset of the plurality of trained machine learning models comprises:
inputting the respective set of characteristics associated with the request into each of the first subset of the plurality of trained machine learning models; and
selecting the content template based on outputs of the first subset of the plurality of trained machine learning models, wherein the content template is associated with an output indicating a highest probability of the respective target performing the target action.

### Claim 21 (depends on 19)

The non-transitory computer readable storage medium of claim 19, further comprising:
determining one of one or more content templates associated with the second subset of the plurality of trained machine learning models; and
for each request in the second group, generating a second content item based on the determined content template.

### Claim 22 (depends on 19)

The non-transitory computer readable storage medium of claim 19, further comprising:
receiving, for each generated content item for each request in the second group, a respective indication as to whether the associated respective target performed a target action responsive to presentation of the respective content item to the respective target;
training the second subset of the plurality of trained machine learning models, wherein each trained machine learning model of the second subset of the plurality of trained machine learning models is trained for a respective content template of the plurality of content templates using a) respective sets of characteristics of targets associated with requests for which the respective content template was selected, b) respective content items generated for the respective requests and c) respective indications associated with the respective content items;
determining, for each trained machine learning model in the second subset, whether the trained machine learning model satisfies the reliability criterion; and
moving those trained machine learning models that satisfy the reliability criterion from the second subset of the plurality of trained machine learning models to the first subset of the plurality of trained machine learning models.

### Claim 23 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein generating the content item based on the content template further comprises:
determining a plurality of components of the content template, each component associated with one or more elements;
for each component of the content template having a plurality of elements, performing the following comprising:
identifying a first subset of a second plurality of trained machine learning models that satisfy another reliability criterion and a second subset of the second plurality of trained machine learning models that fail to satisfy the another reliability criterion, wherein each trained machine learning model of the second plurality of trained machine learning models is associated with a respective element of a respective component of the content template and is trained to output a probability that a target associated with an input set of characteristics would perform a target action responsive to being presented with a content item including the respective element associated with the respective trained machine learning model of the second plurality of trained machine learning models;
assigning the request to a third group or a fourth group;
for each request in the third group, performing the following comprising:
inputting the respective set of characteristics associated with the request into each of the first subset of the second plurality of trained machine learning models; and
selecting an element to be included in the content item for the respective component of the content template, based on outputs of the first subset of the second plurality of trained machine learning models; and
for each request in the fourth group, selecting an element to be included in the content item for the respective component of the content template, based on an element associated with one of the second subset of the second plurality of trained machine learning models;
for each component of the content template having one element, selecting the one element to be included in the content item for the respective component of the content template; and
generating a content item comprising the selected elements associated with the plurality of components of the content template.

### Claim 24 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein a trained machine learning model of the plurality of trained machine learning models corresponds to a machine learning model that is associated with the content template and that is trained using training data comprising training inputs and training outputs,
the training inputs including a plurality set of characteristics associated with targets, and
the training outputs including a plurality of indications as to whether corresponding target actions were performed by the targets responsive to being presented with content items based on the content template.

### Claim 25 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein the reliability criterion is associated with at least one of a mean or a standard deviation of an area under receiver operating characteristic (ROC) curve determined for a respective trained machine learning model of the plurality of trained machine learning models using one or more sets of validation data, the validation data comprising validation input data and validation output data, wherein the validation input data includes another set of characteristics associated with another target and the validation output data includes an indication of whether or not another target action was performed.

### Claim 26 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein assigning the request to the first group or the second group comprises:
determining the ratio of the number of trained machine learning models in the first group to the number of trained machine learning models the second group;
determining a number of requests to assign to the first group and another number of requests to assign to the second group based on the ratio; and
assigning the request to the first group or the second group based on the number of requests to assign to the first group and the another number of requests to assign to the second group.