# US 2020/0401634 A1 · Estimating performance gains for content item delivery using performance replay > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401634 A1 - Document ID: US20200401634A1 - Status: Published application - Kind: A1 - Application number: 16/447,465 - Filed: 2019-06-20 - Published: 2020-12-24 - Earliest claimed: 2019-06-20 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Qing Duan; Jianqiang Shen; Wen Pu - Web page: https://patentgrep.com/patent/US20200401634A1 ## CPC - G06F 16/957: Browsing optimisation, e.g. caching or content distillation - G06F 16/9035: Filtering based on additional data, e.g. user or group profiles - G06F 16/958: Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking - G06Q 30/0246: {Traffic} - G06Q 30/0275: {Auctions} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Technologies for determining performance gains for content item delivery based on modifications to content item selection parameters are provided. The disclosed techniques include implementing a multi-objective optimization model for content item selection using a value for a parameter. The model generates a first plurality of scores and a first ranking for content items. Subset of content items is selected for delivery based on the first ranking. New values for the parameter are identified and for each new value, the content item selection event is replayed. A second plurality of scores and a second ranking is generated for the content items, where the second ranking is different from the first ranking. A third plurality of scores and a third ranking is generated where the third ranking matches the second ranking. A set of gains is calculated for each new value, where each gain corresponds to a different objective of the model. ## Claims ### Claim 1 A method comprising: maintaining a plurality of candidate content items that have a first plurality of scores assigned using a multi-objective optimization model that relies on a particular value for a particular parameter and that are ranked according to a first ranking based on the first plurality of scores; for each possible value in a set of possible values for the particular parameter of the multi-objective optimization model, replaying a content item selection event that involved the multi-objective optimization model with the particular value for the particular parameter, wherein replaying comprises: using the multi-objective optimization model, that relies on said each possible value for the particular parameter, to generate a second plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the second plurality of scores, generating a second ranking of the plurality of candidate content items, wherein the second ranking is different than the first ranking; using the multi-objective optimization model, that relies on said each possible value for the particular parameter, to generate, based on the second ranking, a third plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the third plurality of scores, generating a third ranking of the plurality of candidate content items; and making a determination of whether the third ranking is different than the second ranking; wherein the method is performed by one or more computing devices. ### Claim 2 (depends on 1) The method of claim 1, further comprising: prior to replaying the content item selection event, conducting the content item selection event, wherein conducting comprises: using the multi-objective optimization model, that relies on the particular value for the particular parameter of the multi-objective optimization model, to generate the first plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the first plurality of scores, generating the first ranking of the plurality of candidate content items; and selecting a subset of the plurality of candidate content items for presentation on a computing device. ### Claim 3 (depends on 1) The method of claim 1, further comprising: upon determining that the third ranking is different than the second ranking, using the multi-objective optimization model to generate, based on the third ranking, a fourth plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the fourth plurality of scores, generating a fourth ranking of the plurality of candidate content items; making a second determination of whether the fourth ranking is different than the third ranking; and based on the second determination, calculating a set of gains, associated with said each possible value, each gain, of the set of gains, corresponding to a different objective of a plurality of objectives. ### Claim 4 (depends on 1) The method of claim 1, wherein using the multi-objective optimization model to generate the third plurality of scores comprises, applying a corresponding second ranking position to each candidate content item of the plurality of candidate content items, wherein each corresponding second ranking position represents a relative position of a corresponding candidate content item within the plurality of candidate content items. ### Claim 5 (depends on 1) The method of claim 1, further comprising, based on the determination that the third ranking is the same as the second ranking, calculating a set of gains, associated with said each possible value, each gain, of the set of gains, corresponding to a different objective of a plurality of objectives. ### Claim 6 (depends on 5) The method of claim 5, wherein calculating the set of gains comprises: for each objective of the plurality of objectives: for each candidate content item of the plurality of candidate content items, determining a gain value, for said candidate content item for said objective, as a function of a positional reward based on a positional rank, from the third ranking, of said candidate content item relative to the plurality of candidate content items; calculating a cumulative gain for said objective by aggregating the gain values of the candidate content items in the plurality of candidate content items; and collecting the cumulative gains of the plurality of objectives to generate the set of gains. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the gain value for said candidate content item for said objective comprises: calculating an ideal gain value that is a function of a size of the plurality of candidate content items; and calculating the gain value, for said candidate content item for said objective, as a function of a positional reward based on a positional rank of said candidate content item relative to the plurality of candidate content items divided by the ideal gain value. ### Claim 8 (depends on 7) The method of claim 7, wherein the ideal gain value is based upon a desired set size for the plurality of candidate content items. ### Claim 9 (depends on 1) The method of claim 1, further comprising: selecting a particular possible value of the set of possible values based upon a set of gains associated with each possible value of the set of possible values; using the multi-objective optimization model, with the particular possible value, to generate a new plurality scores for the plurality of candidate content items; based on the new plurality of scores, generating a new ranking of the plurality of candidate content items; and selecting a new subset of the plurality of candidate content items, based on the new ranking, for presentation on a computing device. ### Claim 10 (depends on 1) The method of claim 1, wherein the particular parameter is a weight parameter for a function comprising one objective of an estimated click through rate and a probability of a user interacting with a candidate content item. ### Claim 11 A computer program product comprising: one or more non-transitory computer-readable storage media comprising instructions which, when executed by one or more processors, cause: maintaining a plurality of candidate content items that have a first plurality of scores assigned using a multi-objective optimization model that relies on a particular value for a particular parameter and that are ranked according to a first ranking based on the first plurality of scores; for each possible value in a set of possible values for the particular parameter of the multi-objective optimization model, replaying a content item selection event that involved the multi-objective optimization model with the particular value for the particular parameter, wherein replaying comprises: using the multi-objective optimization model, that relies on said each possible value for the particular parameter, to generate a second plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the second plurality of scores, generating a second ranking of the plurality of candidate content items, wherein the second ranking is different than the first ranking; using the multi-objective optimization model, that relies on said each possible value for the particular parameter, to generate, based on the second ranking, a third plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the third plurality of scores, generating a third ranking of the plurality of candidate content items; and making a determination of whether the third ranking is different than the second ranking. ### Claim 12 (depends on 11) The computer program product of claim 11, wherein the one or more non-transitory computer-readable storage media comprises further instructions which, when executed by the one or more processors, cause: prior to replaying the content item selection event, conducting the content item selection event, wherein conducting comprises: using the multi-objective optimization model, that relies on the particular value for the particular parameter of the multi-objective optimization model, to generate the first plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the first plurality of scores, generating the first ranking of the plurality of candidate content items; and selecting a subset of the plurality of candidate content items for presentation on a computing device. ### Claim 13 (depends on 11) The computer program product of claim 11, wherein the one or more non-transitory computer-readable storage media comprises further instructions which, when executed by the one or more processors, cause: upon determining that the third ranking is different than the second ranking, using the multi-objective optimization model, to generate, based on the third ranking, a fourth plurality of scores, each for a different candidate content item of the plurality of candidate content items; based on the fourth plurality of scores, generating a fourth ranking of the plurality of candidate content items; making a second determination of whether the fourth ranking is different than the third ranking; and based on the second determination, calculating a set of gains, associated with said each possible value, each gain, of the set of gains, corresponding to a different objective of a plurality of objectives. ### Claim 14 (depends on 11) The computer program product of claim 11, wherein using the multi-objective optimization model the third plurality of scores comprises, applying a corresponding second ranking position to each candidate content item of the plurality of candidate content items, wherein each corresponding second ranking position represents a relative position of a corresponding candidate content item within the plurality of candidate content items. ### Claim 15 (depends on 11) The computer program product of claim 11, wherein the one or more non-transitory computer-readable storage media comprises further instructions which, when executed by the one or more processors, cause: based on the determination that the third ranking is the same as the second ranking, calculating a set of gains, associated with said each possible value, each gain, of the set of gains, corresponding to a different objective of a plurality of objectives. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein calculating the set of gains comprises: for each objective of the plurality of objectives: for each candidate content item of the plurality of candidate content items, determining a gain value, for said candidate content item for said objective, as a function of a positional reward based on a positional rank, from the third ranking, of said candidate content item relative to the plurality of candidate content items; calculating a cumulative gain for said objective by aggregating the gain values of each of the candidate content items in the plurality of candidate content items; and collecting the cumulative gains of the plurality of objectives to generate the set of gains. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein determining the gain value for said candidate content item for said objective comprises: calculating an ideal gain value that is a function of a size of the plurality of candidate content items; and calculating the gain value, for said candidate content item for said objective, as a function of a positional reward based on a positional rank of said candidate content item relative to the plurality of candidate content items divided by the ideal gain value. ### Claim 18 (depends on 17) The computer program product of claim 17, wherein the ideal gain value is based upon a desired set size for the plurality of candidate content items. ### Claim 19 (depends on 11) The computer program product of claim 11, wherein the one or more non-transitory computer-readable storage media comprises further instructions which, when executed by the one or more processors, cause: selecting a particular possible value of a set of possible values based upon a set of gains associated with each possible value of the set of possible values; using the multi-objective optimization model, with the particular possible value, to generate a new plurality scores for the plurality of candidate content items; based on the new plurality of scores, generating a new ranking of the plurality of candidate content items; and selecting a new subset of the plurality of candidate content items, based on the new ranking, for presentation on a computing device. ### Claim 20 (depends on 11) The computer program product of claim 11, wherein the particular parameter is a weight parameter for a function comprising one objective of an estimated click through rate and a probability of a user interacting with a candidate content item.