Offline evaluation of ranking functions

A method for offline evaluation of ranking function performance, the method comprising:

• receiving production log data and pre-processing the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different search engine results pages (SERPs) from a search engine in response to a same query issued by a same user in different impressions, each impression of the different impressions being an instance of a SERP selected by a particular ranking function in response to the same query by the same user;
• performing offline estimation of online metrics for a plurality of ranking functions using the exploration data set;
• evaluating a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function;
• detecting that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function;
• and
• displaying a selected SERP on a display module based on the preferred ranking function;
• selecting the same query from a query distribution, the query being one of a plurality of queries by a plurality of users, each query of the plurality of queries having a corresponding SERP;
• and
• evaluating, by the offline estimation, a reward function to compute a reward obtained by executing the ranking function to evaluate each query of the plurality of queries to obtain the corresponding SERP.

One or more computer-readable memory storage devices for storing computer-readable instructions that, when executed by one or more processing devices, instruct offline evaluation of ranking function performance, the computer-readable instructions comprising code to:

• receive production log data and pre-process the production log data to generate an exploration data set, wherein the production log data comprises continuous updates of a ranking function associated with different search engine results pages (SERPs) from a search engine in response to a same query issued by a same user in different impressions, each impression of the different impressions being an instance of a SERP selected by a particular ranking function in response to the same query by the same user;
• perform offline estimation of online metrics for a plurality of ranking functions using the exploration data set;
• evaluate a proposed ranking function by comparing the estimated online metrics to a set of baseline metrics of a baseline ranking function;
• detect that the proposed ranking function is a preferred ranking function having a higher detected metric score than the baseline ranking function based on a comparison of the estimated online metrics with the baseline metrics of the baseline ranking function;
• and
• display a selected SERP on a display module based on the preferred ranking function, wherein
• the same query is one of a plurality of queries by a plurality of users selected from a query distribution, each query of the plurality of queries having a corresponding SERP, and
• the offline estimation evaluates a reward function to compute a reward obtained by executing the ranking function to evaluate each query of the plurality of queries to obtain the corresponding SERP.