# US 2019/0171766 A1 · Multi-stage ranking optimization for selecting content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171766 A1
- Document ID: US20190171766A1
- Status: Published application
- Kind: A1
- Application number: 15/829,843
- Filed: 2017-12-01
- Published: 2019-06-06
- Earliest claimed: 2017-12-01
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Tianshi Gao; Pengjun Pei; Bingqing Wang
- Web page: https://patentgrep.com/patent/US20190171766A1

## CPC

- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 16/24542: {Plan optimisation}
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/40: Business processes related to social networking or social networking services
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

To present one or more content to users of an online system, the online system identifies a content evaluation pipeline including an order of a plurality of stages having one or more computer models for evaluating a likelihood of user interaction with a content item. The content evaluation pipeline selects a decreasing number of content items, from each stage of the order, according to the order of the stages in the order. The online system identifies a set of candidate modifications to one or more operational parameters of the content evaluation pipeline. For each candidate modification, the online system determines a compute time value and a content selection value. For a given amount of compute time, the online system optimizes the one or more operational parameters based on the determined content time value and the determined content selection value to increase the content selection value of the content evaluation pipeline.

## Claims

### Claim 1

A computer-implemented method performed by an online system, the method comprising:
identifying a content evaluation pipeline comprising an order of a plurality of stages having one or more computer models for evaluating a likelihood of user interaction with a content item and the content evaluation pipeline selecting a decreasing number of content items, from each stage of the order, according to the order of the stages in the order;
identifying a set of candidate modifications to modify one or more operational parameters of the content evaluation pipeline;
for each candidate modification, determining a compute time value and a content selection value; and
for a given amount of compute time, optimizing the one or more operational parameters based in part on the determined compute time values and the content selection values to increase the content selection value of the content evaluation pipeline.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the set of computer models comprise a plurality of machine learning models, and a plurality of deep learning models.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the operational parameters comprises the number of stages of a content evaluation pipeline, the number of computer models included in a stage, the number of layers of a computer model, the number of features of a computer model, the number of nodes of a computer model, or a type of a computer model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein optimizing the one or more operational parameters comprises:
selecting a first candidate modification of the set to reduce a complexity of a first computer model that evaluates a less-valuable objective;
selecting a second candidate modification of the set to increase a complexity of a second computer model that evaluates a more-valuable objective; and
applying the first candidate modification and the second candidate modification to the content evaluation pipeline.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein optimizing the one or more operational parameters is further based in part on various selection characteristics, wherein increasing a complexity of the content evaluation pipeline for some selection characteristics that are likely to provide high content selection values, and reducing the complexity of the content evaluation pipeline for other selection characteristics that are likely to provide low content selection values.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein optimizing the one or more operational parameters is further based in part on marginal content selection values relative to marginal compute time, wherein a large marginal content selection value relative to marginal compute time changes a complexity of the content evaluation pipeline more than a small marginal content selection value relative to marginal compute time.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising determine the marginal content selection value relative to marginal compute time, the determining comprises:
determining a difference of content selection values between an initial content evaluation pipeline and the content evaluation pipeline modified by a candidate modification, as the marginal content selection value;
determining a difference of compute time values between the initial content evaluation pipeline and the content evaluation pipeline modified by the candidate modification, as the marginal compute time value; and
dividing the marginal content selection value by the marginal compute time value.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein determining the content selection value comprises:
determining a prediction likelihood indicating that how likely a user will interact with the content; and
multiplying a bid amount with the determined prediction likelihood.

### Claim 9

A non-transitory computer-readable medium comprising computer program instructions that when executed by a computer processor of an online system causes the processor to perform steps comprising:
identifying a content evaluation pipeline comprising an order of a plurality of stages having one or more computer models for evaluating a likelihood of user interaction with a content item and the content evaluation pipeline selecting a decreasing number of content items, from each stage of the order, according to the order of the stages in the order;
identifying a set of candidate modifications to modify one or more operational parameters of the content evaluation pipeline;
for each candidate modification, determining a compute time value and a content selection value; and
for a given amount of compute time, optimizing the one or more operational parameters based in part on the determined compute time values and the content selection values to increase the content selection value of the content evaluation pipeline.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the set of computer models comprise a plurality of machine learning models, and a plurality of deep learning models.

### Claim 11 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the operational parameters comprises the number of stages of a content evaluation pipeline, the number of computer models included in a stage, the number of layers of a computer model, the number of features of a computer model, the number of nodes of a computer model, or a type of a computer model.

### Claim 12 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein optimizing the one or more operational parameters comprises:
selecting a first candidate modification of the set to reduce a complexity of a first computer model that evaluates a less-valuable objective;
selecting a second candidate modification of the set to increase a complexity of a second computer model that evaluates a more-valuable objective; and
applying the first candidate modification and the second candidate modification to the content evaluation pipeline.

### Claim 13 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein optimizing the one or more operational parameters is further based in part on various selection characteristics, wherein increasing a complexity of the content evaluation pipeline for some selection characteristics that are likely to provide high content selection values, and reducing the complexity of the content evaluation pipeline for other selection characteristics that are likely to provide low content selection values.

### Claim 14 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein optimizing the one or more operational parameters is further based in part on marginal content selection values relative to marginal compute time, wherein a large marginal content selection value relative to marginal compute time changes a complexity of the content evaluation pipeline more than a small marginal content selection value relative to marginal compute time.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, further comprising determine the marginal content selection value relative to marginal compute time, the determining comprises:
determining a difference of content selection values between an initial content evaluation pipeline and the content evaluation pipeline modified by a candidate modification, as the marginal content selection value;
determining a difference of compute time values between the initial content evaluation pipeline and the content evaluation pipeline modified by the candidate modification, as the marginal compute time value; and
dividing the marginal content selection value by the marginal compute time value.

### Claim 16 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein determining the content selection value comprises:
determining a prediction likelihood indicating that how likely a user will interact with the content; and
multiplying a bid amount with the determined prediction likelihood.

### Claim 17

A system comprising:
a processor; and
a non-transitory computer-readable medium comprising computer program instructions that when executed by the processor of an online system causes the processor to perform steps comprising:
identifying a content evaluation pipeline comprising an order of a plurality of stages having one or more computer models for evaluating a likelihood of user interaction with a content item and the content evaluation pipeline selecting a decreasing number of content items, from each stage of the order, according to the order of the stages in the order;
identifying a set of candidate modifications to modify one or more operational parameters of the content evaluation pipeline;
for each candidate modification, determining a compute time value and a content selection value; and
for a given amount of compute time, optimizing the one or more operational parameters based in part on the determined compute time values and the content selection values to increase the content selection value of the content evaluation pipeline.

### Claim 18 (depends on 17)

The system of claim 17, wherein the set of computer models comprise a plurality of machine learning models, and a plurality of deep learning models.

### Claim 19 (depends on 17)

The system of claim 17, wherein the operational parameters comprises the number of stages of a content evaluation pipeline, the number of computer models included in a stage, the number of layers of a computer model, the number of features of a computer model, the number of nodes of a computer model, or a type of a computer model.

### Claim 20 (depends on 17)

The system of claim 17, wherein optimizing the one or more operational parameters comprises:
selecting a first candidate modification of the set to reduce a complexity of a first computer model that evaluates a less-valuable objective;
selecting a second candidate modification of the set to increase a complexity of a second computer model that evaluates a more-valuable objective; and
applying the first candidate modification and the second candidate modification to the content evaluation pipeline.
