Framework for evaluating web search scoring functions

A system comprising:

• one or more processors;
• and
• one or more memory devices configured to cause the one or more processors to perform operations comprising:
• receiving a query;
• selecting a first candidate scoring function and a second candidate scoring function from a group of at least three candidate scoring functions;
• presenting search results, which are responsive to the query, ordered according to scores from the first candidate scoring function;
• presenting the search results, which are responsive to the query, ordered according to scores from the second candidate scoring function;
• receiving user input selecting one of the first and second candidate scoring functions over the other;
• and
• using the user input to inform future selections of pairs of candidate scoring functions from the group of candidate scoring functions.

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform actions comprising:
• receiving a plurality of queries sequentially, each query being received from a respective user;
• and
• performing the following operations with respect to each query and respective user:
• selecting a pair of candidate scoring functions from a group of at least three candidate scoring functions, the pair including a first candidate scoring function and a second candidate scoring function, the selecting being based at least in part on a ranking of the candidate scoring functions in the group;
• presenting search results, which are responsive to the query, ordered according to scores from the first candidate scoring function, to the respective user;
• presenting the search results, which are responsive to the query, ordered according to scores from the second candidate scoring function, to the respective user;
• receiving user input, from the respective user, selecting one of the first and second candidate scoring functions over the other;
• and
• using the user input to update the ranking of the candidate scoring functions in the group.