Machine-assisted search preference evaluation

A system that evaluates search preferences, comprising:

• at least one processor;
• and
• memory that comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:
• receiving a search query, a first search result list returned by a first ranker system responsive to the search query, and a second search result list returned by a second ranker system responsive to the search query;
• generating a prediction of a search preference judgment of the first search result list and the second search result list for the search query;
• and
• transmitting the search query, the first search result list, and the second search result list to an annotator computing device for manual judgment by a judge when the prediction of the search preference judgment specifies:
• the first search result list and the second search result list for the search query are predicted to have a quality difference;
• and
• a confidence level that one of the first search result list or the second search result list is preferred is predicted to be equal to or below a threshold value;
• wherein the search query, a first search result list, and the second search result list are otherwise inhibited from being transmitted for manual judgment.

A computer-readable storage medium including computer-executable instructions that, when executed by a processor, cause the processor to perform acts including:

• receiving a set of search queries and search result list pairs, wherein search result lists in the search result list pairs are respectively returned by a first ranker system and a second ranker system responsive to the search queries;
• receiving manually judged side-by-side search preference judgments of the set of the search queries and the search result list pairs;
• determining values of features of the search queries and the search result list pairs;
• and
• training one or more classifiers based upon the manually judged side-by-side search preference judgments of the set of the search queries and the search result list pairs and the values of the features of the search queries and the search result list pairs, wherein the one or more classifiers are trained to generate predicted side-by-side search preference judgments and confidence levels for the predicted side-by-side search preference judgments.