# US 2025/0094432 A1 · Offline evaluation of ranking functions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094432 A1
- Document ID: US20250094432A1
- Status: Published application
- Kind: A1
- Application number: 18/965,948
- Filed: 2024-12-02
- Published: 2025-03-20
- Earliest claimed: 2014-11-21
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Lihong LI; Jinyoung KIM; Imed ZITOUNI
- Web page: https://patentgrep.com/patent/US20250094432A1

## CPC

- G06F 16/24578: {using ranking}
- G06Q 30/00: Commerce
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

An example system includes a processor to receive production log data and pre-process the production log data to generate an exploration data set. The production log data includes continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions. The processor can perform offline estimation of online metrics for ranking functions using the exploration data set, evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function, detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function, and execute an action on an information retrieval system based on the preferred ranking function.

## Claims

### Claim 1

A system for offline evaluation of ranking functions, comprising a processor to:
select a query issued by a user from a query distribution including a plurality of queries by a plurality of users, each query of the plurality of queries having corresponding search engine results selected by a ranking function;
receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a particular ranking function associated with different search engine results from a search engine in response to the selected query issued by the user in different impressions, each impression of the different impressions being an instance of search engine results selected by the particular ranking function in response to the selected query issued by the user;
perform offline estimation of metrics for a plurality of ranking functions using the exploration data set;
evaluate a proposed ranking function by comparing the estimated metrics to a set of baseline metrics of a baseline ranking function;
detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated metrics with the baseline metrics of the baseline ranking function; and
output a selected search engine result based on the preferred ranking function.

### Claim 2 (depends on 1)

The system of claim 1, wherein the processor is to output the selected search engine result based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor is to pre-process the production log data by aggregating the production log data by query, by search engine result, by probability of action, and by reward value.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processor is to use approximate search engine result matching of rankings to compute the estimated metrics.

### Claim 5 (depends on 4)

The system of claim 4, wherein the approximate search engine result matching comprises comparing a predetermined number of higher-ranked results for each search engine result generated by a respective ranking function.

### Claim 6 (depends on 1)

The system of claim 1, wherein the metrics comprise at least one of a click-through rate (CTR), a time to click on a search engine result, and a mean reciprocal of click positions.

### Claim 7 (depends on 1)

The system of claim 1, wherein the preferred ranking function is used to output the selected search engine result in response to detecting that a quality of the preferred ranking function exceeds a threshold during a test.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is to:
generate query results with the proposed ranking function as a test of the proposed ranking function; and
output the generated query results.

### Claim 9 (depends on 1)

The system of claim 1, wherein the processor is to perform the offline evaluation of the metrics for the plurality of ranking functions at least in part at a contextual bandit model.

### Claim 10 (depends on 1)

The system of claim 1, wherein the processor is to:
calculate a delta metric score between the preferred ranking function and the baseline ranking function; and
detect that the delta metric score indicates that the preferred ranking function has a higher estimated online metric than the online metric of the baseline ranking function.

### Claim 11 (depends on 1)

The system of claim 1, wherein the processor is to detect whether a first ranking function from the plurality of ranking functions has a relevance score that is within a predetermined range of the relevance score of a second ranking function, higher than the predetermined range, or lower than the predetermined range.

### Claim 12

A method for offline evaluation of ranking function performance, the method comprising:
selecting a query issued by a user from a query distribution including a plurality of queries by a plurality of users, each query of the plurality of queries having corresponding search engine results selected by a ranking function;
receiving production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a particular ranking function associated with different search engine results from a search engine in response to the selected query issued by the user in different impressions, each impression of the different impressions being an instance of search engine results selected by the particular ranking function in response to the selected query issued by the user;
performing offline estimation of metrics for a plurality of ranking functions using the exploration data set;
evaluating a proposed ranking function by comparing the estimated metrics to a set of baseline metrics of a baseline ranking function;
detecting that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated metrics with the baseline metrics of the baseline ranking function; and
output a selected search engine result based on the preferred ranking function.

### Claim 13 (depends on 12)

The method of claim 12, further comprising outputting the selected search engine result based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold.

### Claim 14 (depends on 12)

The method of claim 12, further comprising pre-processing the production log data by aggregating the production log data by query, by search engine result, by probability of action, and by reward value.

### Claim 15 (depends on 12)

The method of claim 12, further comprising using approximate search engine result matching of rankings to compute the estimated metrics.

### Claim 16 (depends on 12)

The method of claim 12, wherein the metrics comprise at least one of a click-through rate (CTR), a time to click on a search engine result, and a mean reciprocal of click positions.

### Claim 17 (depends on 12)

The method of claim 12, wherein the preferred ranking function is used to output the selected search engine result in response to detecting that a quality of the preferred ranking function exceeds a threshold during a test.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
generating query results with the proposed ranking function as a test of the proposed ranking function; and
outputting the generated query results.

### Claim 19 (depends on 12)

The method of claim 12, further comprising performing the offline evaluation of the metrics for the plurality of ranking functions at least in part at a contextual bandit model.

### Claim 20

One or more computer-readable memory storage devices for storing computer-readable instructions that, when executed by one or more processing devices, instruct offline evaluation of ranking function performance, the computer-readable instructions comprising code to:
select a query issued by a user from a query distribution including a plurality of queries by a plurality of users, each query of the plurality of queries having corresponding search engine results selected by a ranking function;
receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a particular ranking function associated with different search engine results from a search engine in response to the selected query issued by the user in different impressions, each impression of the different impressions being an instance of search engine results selected by the particular ranking function in response to the selected query issued by the user;
perform offline estimation of metrics for a plurality of ranking functions using the exploration data set;
evaluate a proposed ranking function by comparing the estimated metrics to a set of baseline metrics of a baseline ranking function;
detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated metrics with the baseline metrics of the baseline ranking function; and
output a selected search engine result based on the preferred ranking function.
