# US 2025/0095007 A1 · Training a model to predict likelihoods of users performing an action after being presented with a content item

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095007 A1
- Document ID: US20250095007A1
- Status: Published application
- Kind: A1
- Application number: 18/967,681
- Filed: 2024-12-04
- Published: 2025-03-20
- Earliest claimed: 2021-06-09
- Assignees: Maplebear Inc.
- Applicants: Maplebear Inc.
- Inventors: Changyao Chen; Peng Qi; Weian Sheng
- Web page: https://patentgrep.com/patent/US20250095007A1

## CPC

- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 20/20: Ensemble learning
- G06Q 30/0206: {Price or cost determination based on market factors}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An online concierge system trains a user interaction model to predict a probability of a user performing an interaction after one or more content items are displayed to the user. This provides a measure of an effect of displaying content items to the user on the user performing one or more interactions. The user interaction model is trained from displaying content items to certain users of the online concierge system and withholding display of the content items to other users of the online concierge system. To train the user interaction model, the user interaction model is applied to labeled examples identifying a user and value based on interactions the user performed after one or more content items were displayed to the user and interactions the user performed when one or more content items were not used.

## Claims

### Claim 1

A computer-implemented method performed at a computer system comprising a processor and a computer-readable medium, the computer-implemented method comprising:
displaying one or more content items to a first user;
measuring a first number of interactions performed by the first user after the one or more content items are displayed to the first user;
measuring a second number of interactions performed by a second user when none of the one or more content items are displayed to the second user;
determining a measured difference between the first number of interactions and the second number of interactions;
accessing a user interaction model that comprises a plurality of layers of a neural network, wherein the user interaction model is trained by:
applying the user interaction model to generate a predicted difference, the predicted difference indicating a difference between a first predicted number of interactions performed by the first user after the one or more content items are displayed to the first user and a second predicted number of the interaction performed by the second user when none of the one or more content items are displayed to the second user, and
backpropagating one or more error terms obtained from one or more loss functions to update a set of parameters of the user interaction model, the backpropagating performed through the neural network and one or more of the error terms based on the measured difference and the predicted difference;
identifying an opportunity to display content items to an additional user;
applying the user interaction model to determine a predicted value of the additional user when a particular content item is displayed to the additional user;
determining, using the predicted value, whether to display the particular content item to the additional user; and
causing, based on determining to display the particular content item to the additional user, the particular content item to be displayed to the additional user.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein measuring the first number of interactions comprises measuring a first average rate at which the first user performs an interaction after the one or more content items are displayed to the first user, and measuring the second number of interactions comprises measuring a second average rate at which the second user performs an interaction after none of the one or more content items are displayed to the second user.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the interactions measured comprise one or more of the following: clicking on a link associated with a content item, performing a search associated with a content item and including, in an order, an item that is presented in a content item.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein training the user interaction model comprises apply a training set to train the user interaction model, the training set comprising measurements of interactions of a plurality of users.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the additional user is not included in the plurality of users in the training set.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the computer system corresponds to an online concierge system that suggests one or more items to be added in an order as content items.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein identifying an opportunity to display content items to an additional user comprises identifying a plurality of candidate content items; and
wherein determining, using the predicted value, whether to display the particular content item to the additional user comprises:
generating a ranking of predicted values for the plurality of candidate content items; and
selecting one of the candidate content items as the particular content item to be displayed based on the ranking.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein measuring the first number of interactions comprises measuring the first number of interactions when the first user is not in a holdout group, and measuring the second number of interactions comprises measuring the second number of interactions when the second user is in a holdout group.

### Claim 9

A non-transitory computer-readable medium configured to store code comprising instructions, wherein the instructions, when executed by one or more processors of a computer system, cause the one or more processors to:
display one or more content items to a first user;
measure a first number of interactions performed by the first user after the one or more content items are displayed to the first user;
measure a second number of interactions performed by a second user when none of the one or more content items are displayed to the second user;
determine a measured difference between the first number of interactions and the second number of interactions;
access a user interaction model that comprises a plurality of layers of a neural network, wherein the user interaction model is trained by:
applying the user interaction model to generate a predicted difference, the predicted difference indicating a difference between a first predicted number of interactions performed by the first user after the one or more content items are displayed to the first user and a second predicted number of the interaction performed by the second user when none of the one or more content items are displayed to the second user, and
backpropagating one or more error terms obtained from one or more loss functions to update a set of parameters of the user interaction model, the backpropagating performed through the neural network and one or more of the error terms based on the measured difference and the predicted difference;
identify an opportunity to display content items to an additional user;
apply the user interaction model to determine a predicted value of the additional user when a particular content item is displayed to the additional user;
determine, using the predicted value, whether to display the particular content item to the additional user; and
cause, based on determining to display the particular content item to the additional user, the particular content item to be displayed to the additional user.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instruction to measure the first number of interactions comprises instructions to measure a first average rate at which the first user performs an interaction after the one or more content items are displayed to the first user, and the instruction to measure the second number of interactions comprises instructions to measure a second average rate at which the second user performs an interaction after none of the one or more content items are displayed to the second user.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the interactions measured comprise one or more of the following: clicking on a link associated with a content item, performing a search associated with a content item and including, in an order, an item that is presented in a content item.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein training the user interaction model further comprises apply a training set to train the user interaction model, the training set comprising measurements of interactions of a plurality of users.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the additional user is not included in the plurality of users in the training set.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the computer system corresponds to an online concierge system that suggests one or more items to be added in an order as content items.

### Claim 15 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instruction to identify an opportunity to display content items to an additional user comprises instructions to identify a plurality of candidate content items; and
wherein the instruction to determine, using the predicted value, whether to display the particular content item to the additional user comprises instructions to:
generate a ranking of predicted values for the plurality of candidate content items; and
select one of the candidate content items as the particular content item to be displayed based on the ranking.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instruction to measure the first number of interactions comprises instructions to measure the first number of interactions when the first user is not in a holdout group, and the instruction to measure the second number of interactions comprises instructions to measure the second number of interactions when the second user is in a holdout group.

### Claim 17

A system comprising:
one or more processors; and
memory configured to store code comprising instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
display one or more content items to a first user;
measure a first number of interactions performed by the first user after the one or more content items are displayed to the first user;
measure a second number of interactions performed by a second user when none of the one or more content items are displayed to the second user;
determine a measured difference between the first number of interactions and the second number of interactions;
access a user interaction model that comprises a plurality of layers of a neural network, wherein the user interaction model is trained by:
applying the user interaction model to generate a predicted difference, the predicted difference indicating a difference between a first predicted number of interactions performed by the first user after the one or more content items are displayed to the first user and a second predicted number of the interaction performed by the second user when none of the one or more content items are displayed to the second user, and
backpropagating one or more error terms obtained from one or more loss functions to update a set of parameters of the user interaction model, the backpropagating performed through the neural network and one or more of the error terms based on the measured difference and the predicted difference;
identify an opportunity to display content items to an additional user;
apply the user interaction model to determine a predicted value of the additional user when a particular content item is displayed to the additional user;
determine, using the predicted value, whether to display the particular content item to the additional user; and
cause, based on determining to display the particular content item to the additional user, the particular content item to be displayed to the additional user.

### Claim 18 (depends on 17)

The system of claim 17, wherein the instruction to measure the first number of interactions comprises instructions to measure a first average rate at which the first user performs an interaction after the one or more content items are displayed to the first user, and the instruction to measure the second number of interactions comprises instructions to measure a second average rate at which the second user performs an interaction after none of the one or more content items are displayed to the second user.

### Claim 19 (depends on 17)

The system of claim 17, wherein the interactions measured comprise one or more of the following: clicking on a link associated with a content item, performing a search associated with a content item and including, in an order, an item that is presented in a content item.

### Claim 20 (depends on 17)

The system of claim 17, wherein training the user interaction model further comprises apply a training set to train the user interaction model, the training set comprising measurements of interactions of a plurality of users.