# US 2022/0012768 A1 · Iteratively improving an advertisement response model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012768 A1
- Document ID: US20220012768A1
- Status: Published application
- Kind: A1
- Application number: 17/465,768
- Filed: 2021-09-02
- Published: 2022-01-13
- Earliest claimed: 2015-12-07
- Assignees: Paypal, Inc.
- Applicants: Paypal, Inc.
- Inventors: EGOR KOBYLKIN
- Web page: https://patentgrep.com/patent/US20220012768A1

## CPC

- G06Q 30/0244: {Optimization}
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06Q 10/067: Enterprise or organisation modelling
- G06Q 30/0254: {based on statistics}
- G06Q 30/0273: Determination of fees for advertising
- G06Q 30/0277: {Online advertisement}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

There are provided systems and methods for iteratively improving an advertisement response model. A payment provider may perform operations that include training an advertisement response model using a training data set. The operations include determining that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold. The operations further include identifying, based on executing the advertisement response model using a target data set that is different from the training data set, one or more units from the target data set for which to run the advertising campaign. The operations also include receiving one or more responses corresponding to a run of the advertising campaign with respect to the identified one or more units from the target data set and updating the training data set based on the one or more responses. The operations further include training an advertisement response model using resulting training data and repeating the operations as long as the accuracy value of the resulting model stays below the threshold or until the increase in the accuracy value with each iteration becomes unprofitable with respect to the costs of acquiring responses from further units from the target dataset.

## Claims

### Claim 1

(canceled)

### Claim 2

A system, comprising:
at least one processor; and
at least one memory storing computer-executable instructions, that in response to execution by the at least one processor, causes the system to perform operations comprising
executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign;
determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold;
calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set;
determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign;
receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and
determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold.

### Claim 3 (depends on 2)

The system of claim 2, wherein the operations further comprise:
determining that the second accuracy value is greater than or equal to the accuracy value threshold;
adjusting the target data set by removing the units included in the subset from the target data set; and
based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.

### Claim 4 (depends on 2)

The system of claim 2, wherein the operations further comprise removing the subset of the target data set from the target data set.

### Claim 5 (depends on 2)

The system of claim 2, wherein determining the subset of the target data set comprises:
ranking each unit of the target data set according to the respective certainty score for each unit of the target data set, wherein the subset includes a predetermined number of units from the target data set that are each associated with a lower respective certainty score than respective certainty scores of remaining units in the target data set that are not included in the sub set.

### Claim 6 (depends on 2)

The system of claim 2, wherein each unit of the subset of the target data set is associated with a certainty score that is less than a certainty score threshold.

### Claim 7 (depends on 2)

The system of claim 2, wherein the operations further comprise generating the first training data set by transmitting advertisements corresponding to the advertising campaign to respective devices of a random sample of a particular population.

### Claim 8 (depends on 2)

The system of claim 2, wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics.

### Claim 9 (depends on 2)

The system of claim 2, wherein the second accuracy value is greater than the first accuracy value.

### Claim 10

A method comprising:
executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign;
determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold;
calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set;
determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign;
receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and
determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
determining that the second accuracy value is greater than or equal to the accuracy value threshold;
adjusting the target data set by removing the units included in the subset from the target data set; and
based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
removing the subset of the target data set from the target data set.

### Claim 13 (depends on 10)

The method of claim 10, wherein determining the subset of the target data set comprises:
ranking each unit of the target data set according to the respective certainty score for each unit of the target data set, wherein the subset includes a predetermined number of units from the target data set that are each associated with a lower respective certainty score than respective certainty scores of remaining units in the target data set that are not included in the subset.

### Claim 14 (depends on 10)

The method of claim 10, wherein each unit of the subset of the target data set is associated with a certainty score that is less than a certainty score threshold.

### Claim 15 (depends on 10)

The method of claim 10, further comprising:
generating the first training data set by transmitting advertisements corresponding to the advertising campaign to respective devices of a random sample of a particular population.

### Claim 16 (depends on 10)

The method of claim 10, wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics.

### Claim 17 (depends on 10)

The method of claim 10, wherein the second accuracy value is greater than the first accuracy value.

### Claim 18

A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
executing an advertisement response model using a first training data set, wherein the first training data set indicates, for each unit included in the training data set, a respective response to an advertising campaign;
determining, based on the executing the advertisement response model using the first training data set, that a first accuracy value corresponding to the advertisement response model is less than a accuracy value threshold;
calculating, based on executing the advertisement response model using a target data set that is different from the first training data set, a respective certainty score corresponding to each unit of the target data set;
determining, based on the respective certainty score corresponding to each unit of the target data set, a subset of the target data set for which to run the advertising campaign;
receiving one or more responses corresponding to a run of the advertising campaign with respect to the subset of the target data set, and generating a second training data set based on the one or more responses; and
determining, based on executing the advertisement response model using the second training data set, whether a second accuracy value corresponding to the advertisement response model is greater than the accuracy value threshold.

### Claim 19 (depends on 18)

The non-transitory machine-readable medium of claim 18, further comprising:
determining that the second accuracy value is greater than or equal to the accuracy value threshold;
adjusting the target data set by removing the units included in the subset from the target data set; and
based on executing the advertisement response model with respect to the adjusted target data set, determining one or more units of the adjusted target data set that have a positive predicted response to the advertising campaign.

### Claim 20 (depends on 18)

The non-transitory machine-readable medium of claim 18, further comprising:
removing the subset of the target data set from the target data set.

### Claim 21 (depends on 18)

The non-transitory machine-readable medium of claim 18, wherein the subset of the target data corresponds to one or more characteristics selected from a group comprising website characteristics, geographic location characteristics, and demographic characteristics.
