# US 12,182,140 B2 · Offline evaluation of ranking functions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,140 B2 - Document ID: US12182140B2 - Status: Granted - Kind: B2 - Application number: 18/176,880 - Filed: 2023-03-01 - Published: 2023-06-29 - Granted: 2024-12-31 - Earliest claimed: 2014-11-21 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Lihong Li; Jinyoung Kim; Imed Zitouni - Web page: https://patentgrep.com/patent/US12182140B2 ## CPC - G06F 16/24578: {using ranking} - G06Q 30/00: Commerce - G06Q 30/02: Marketing; Price estimation or determination; Fundraising - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract An example system includes a processor to receive production log data and pre-process the production log data to generate an exploration data set. The production log data includes continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions. The processor can perform offline estimation of online metrics for ranking functions using the exploration data set, evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function, detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function, and execute an action on an information retrieval system based on the preferred ranking function. ## Claims ### Claim 1 A system for offline evaluation of ranking functions, comprising a processor to: receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different search engine results pages (SERPs) from a search engine in response to a same query issued by a same user in different impressions, each impression of the different impressions being an instance of a SERP selected by a particular ranking function in response to the same query by the same user; perform offline estimation of online metrics for a plurality of ranking functions using the exploration data set; evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function; detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function; and display a selected SERP on a display module based on the preferred ranking function, wherein the same query is one of a plurality of queries by a plurality of users selected from a query distribution, each query of the plurality of queries having a corresponding SERP, and the offline estimation evaluates a reward function to compute a reward obtained by executing the ranking function to evaluate each query of the plurality of queries to obtain the corresponding SERP. ### Claim 2 (depends on 1) The system of claim 1, wherein the processor is to display the selected SERP on the display module based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold. ### Claim 3 (depends on 1) The system of claim 1, wherein the processor is to pre-process the production log data by aggregating the production log data by query, by SERP, by probability of action, and by reward value. ### Claim 4 (depends on 1) The system of claim 1, wherein the processor is to use approximate SERP matching of rankings to estimate online metrics. ### Claim 5 (depends on 4) The system of claim 4, wherein approximate SERP matching comprises comparing a predetermined number of higher-ranked results for each SERP generated by a respective ranking function. ### Claim 6 (depends on 1) The system of claim 1, wherein the online metrics comprise at least one of a click-through rate (CTR), a time to click on a SERP, and a mean reciprocal of click positions. ### Claim 7 (depends on 1) The system of claim 1, wherein the preferred ranking function is to be used to display the selected SERP on the display module in response to detecting that a quality of the preferred ranking function exceeds a threshold during a test. ### Claim 8 (depends on 1) The system of claim 1, wherein the processor is to: generate query results with the proposed ranking function as a test of the proposed ranking function; and display the generated query results. ### Claim 9 A method for offline evaluation of ranking function performance, the method comprising: receiving production log data and pre-processing the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different search engine results pages (SERPs) from a search engine in response to a same query issued by a same user in different impressions, each impression of the different impressions being an instance of a SERP selected by a particular ranking function in response to the same query by the same user; performing offline estimation of online metrics for a plurality of ranking functions using the exploration data set; evaluating a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function; detecting that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function; and displaying a selected SERP on a display module based on the preferred ranking function; selecting the same query from a query distribution, the query being one of a plurality of queries by a plurality of users, each query of the plurality of queries having a corresponding SERP; and evaluating, by the offline estimation, a reward function to compute a reward obtained by executing the ranking function to evaluate each query of the plurality of queries to obtain the corresponding SERP. ### Claim 10 (depends on 9) The method of claim 9, further comprising approximately matching SERPs in the exploration data set. ### Claim 11 (depends on 10) The method of claim 10, further comprising approximately matching SERPs in the exploration data set by matching a predetermined number of higher-ranked results for each SERP. ### Claim 12 (depends on 9) The method of claim 9, further comprising detecting whether a first ranking function from the plurality of ranking functions has a relevance score that is within a predetermined range of the relevance score of a second ranking function, higher than the predetermined range, or lower than the predetermined range. ### Claim 13 (depends on 9) The method of claim 9, further comprising calculating a confidence score that indicates a level of certainty of results of the comparison and displaying the confidence score with an associated comparison result. ### Claim 14 (depends on 9) The method of claim 9, wherein displaying the selected SERP on the display module based on the preferred ranking function is performed in response to detecting that a quality of the preferred ranking function exceeds a threshold. ### Claim 15 (depends on 9) The method of claim 9, preprocessing the production log data further comprising aggregating the production log data by query, by SERP, by probability of action and by reward value. ### Claim 16 (depends on 9) The method of claim 9, further comprising sending the preferred ranking function to a server during a testing process. ### Claim 17 (depends on 16) The method of claim 16, further comprising testing the preferred ranking function on users via the server during the testing process. ### Claim 18 One or more computer-readable memory storage devices for storing computer-readable instructions that, when executed by one or more processing devices, instruct offline evaluation of ranking function performance, the computer-readable instructions comprising code to: receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different search engine results pages (SERPs) from a search engine in response to a same query issued by a same user in different impressions, each impression of the different impressions being an instance of a SERP selected by a particular ranking function in response to the same query by the same user; perform offline estimation of online metrics for a plurality of ranking functions using the exploration data set; evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function; detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function; and display a selected SERP on a display module based on the preferred ranking function, wherein the same query is one of a plurality of queries by a plurality of users selected from a query distribution, each query of the plurality of queries having a corresponding SERP, and the offline estimation evaluates a reward function to compute a reward obtained by executing the ranking function to evaluate each query of the plurality of queries to obtain the corresponding SERP. ### Claim 19 (depends on 18) The one or more computer-readable memory storage devices of claim 18, further comprising code to: calculate a delta metric score between the preferred ranking function and the baseline ranking function; and detect that the delta metric score indicates that the preferred ranking function has a higher estimated online metric than the online metric of the baseline ranking function.