Offline evaluation of ranking functions

A method for offline evaluation of ranking function performance, the method comprising:

• receiving production log data;
• pre-processing the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;
• performing an offline estimation of online metrics using the exploration data set for a plurality of ranking functions, wherein a proposed ranking function is received and tested offline prior to being tested online on users;
• comparing the plurality of ranking functions based on the estimated online metrics to generate comparison results;
• identifying one or more preferred ranking functions having a higher detected metric score than the baseline ranking function based on the comparison results;
• generating query results with the preferred ranking function during a testing process;
• and
• displaying the generated query results.

One or more computer-readable memory storage devices for storing computer readable instructions that, when executed by one or more processing devices, instruct the offline evaluation of ranking function performance, the computer-readable instructions comprising code to:

• receive production log data;
• preprocess the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different actions for a same query issued by a same user in different impressions;
• perform offline estimates of online metrics for a ranking function based at least in part on the exploration data set and an approximate action matching process, wherein a proposed ranking function is received and tested offline prior to being tested online on users;
• detect that the ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with baseline ranking function metrics;
• execute an action on an information retrieval system based on the preferred ranking function in response to detecting that a quality of the preferred ranking function exceeds a threshold during a testing process;
• and
• display generated results from the information retrieval system.