# US 8,572,075 B1 · Framework for evaluating web search scoring functions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,572,075 B1 - Document ID: US8572075B1 - Status: Granted - Kind: B1 - Application number: 13/296,099 - Filed: 2011-11-14 - Granted: 2013-10-29 - Earliest claimed: 2009-07-23 - Assignees: Google Inc. - Applicants: Misha Zatsman; Paul G. Haahr; Matthew D. Cutts; Yonghui Wu - Inventors: Misha Zatsman; Paul G. Haahr; Matthew D. Cutts; Yonghui Wu - Web page: https://patentgrep.com/patent/US8572075B1 ## CPC - G06F 16/9538: Presentation of query results - G06F 16/951: Indexing; Web crawling techniques - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer program products, for testing web search scoring functions. A query is received. A first and a second scoring function are selected by receiving search results responsive to the query; applying candidate scoring functions to the search results to determine scores for the search results for each candidate scoring function; identifying pairs of the candidate scoring functions, and calculating a diversity score for each of the pairs. A pair of candidate scoring functions is chosen from the one or more pairs of candidate scoring functions based on the diversity scores, and the alpha function is selected as the first scoring function and the beta function is selected as the second scoring function. The plurality of search results are presented in an order according to scores from the first scoring function and are presented in an order according to scores from the second scoring function. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a query at a computer; selecting a first candidate scoring function and a second candidate scoring function from a group of at least three candidate scoring functions; presenting search results, which are responsive to the query, ordered according to scores from the first candidate scoring function; presenting the search results, which are responsive to the query, ordered according to scores from the second candidate scoring function; receiving user input selecting one of the first and second candidate scoring functions over the other; and using the user input to inform future selections of pairs of candidate scoring functions from the group of candidate scoring functions. ### Claim 2 (depends on 1) The method of claim 1, further comprising: classifying the query; and using the classification to select the group of candidate scoring functions from a plurality of available scoring functions. ### Claim 3 (depends on 1) The method of claim 1, further comprising selecting the group of candidate scoring functions from a plurality of available scoring functions based on a respective score associated with each scoring function. ### Claim 4 (depends on 1) The method of claim 1, wherein using the user input to inform future selections of pairs of candidate scoring functions from the group of candidate scoring functions comprises: generating a ranking for the candidate scoring functions in the group based at least in part on the user input; and selecting, for evaluation by one or more other users, one or more candidate scoring functions in the group based at least in part on the ranking for the candidate scoring functions in the group. ### Claim 5 (depends on 4) The method of claim 4, comprising: applying each respective candidate scoring function of the group to the search results to determine respective scores for the search results for the respective candidate scoring function; and comparing results located in same positions in respective sets of the search results, ordered according to respective candidate scoring functions, for each of the pairs of candidate scoring functions; wherein the selecting is based on output of the comparing. ### Claim 6 (depends on 5) The method of claim 5, wherein the comparing comprises comparing results in first through tenth positions in the respective sets of the search results. ### Claim 7 (depends on 5) The method of claim 5, wherein the applying comprises providing signals including information about the query and each result in the search results to each respective candidate scoring function. ### Claim 8 (depends on 4) The method of claim 4, wherein generating a ranking for the candidate scoring functions in the group based at least in part on the user input comprises: identifying a user and an associated candidate scoring function from the group of candidate scoring functions, the user having provided the user input, and the user being associated with the associated candidate scoring function; and determining a number of inputs received from different users, including the identified user, associated with the associated candidate scoring function; and generating the ranking for the candidate scoring functions in the group based at least in part on the number of inputs received from the different users. ### Claim 9 (depends on 4) The method of claim 4, wherein generating a ranking for the candidate scoring functions in the group based at least in part on the user input comprises: adding the user input to a set of multiple pair-wise user comparisons of scoring functions; and generating the ranking for the candidate scoring functions in the group based at least in part on the set of multiple pair-wise user comparisons. ### Claim 10 (depends on 9) The method of claim 9, wherein the search results ordered according to scores from the first candidate scoring function constitute a first order, and the method comprises: receiving a notification that the first candidate scoring function has been modified; applying the modified first scoring function to the search results to determine modified scores for the search results and to generate a second order for the search results according to the modified scores; determining that the first order and the second order are different; and modifying the ranking for the candidate scoring functions in the group so that the modified ranking for the candidate scoring functions in the group does not reflect the user input. ### Claim 11 (depends on 4) The method of claim 4, further comprising: receiving a second query; selecting a third candidate scoring function and a fourth candidate scoring function from a top number of scoring functions in the ranking for the candidate scoring functions in the group; presenting search results, which are responsive to the second query, ordered according to scores from the third candidate scoring function; presenting the search results, which are responsive to the second query, ordered according to scores from the fourth candidate scoring function; receiving second user input selecting one of the third and fourth candidate scoring functions over the other; and updating the ranking for the candidate scoring functions in the group based at least in part on the second user input. ### Claim 12 A system comprising: one or more processors; and one or more memory devices configured to cause the one or more processors to perform operations comprising: receiving a query; selecting a first candidate scoring function and a second candidate scoring function from a group of at least three candidate scoring functions; presenting search results, which are responsive to the query, ordered according to scores from the first candidate scoring function; presenting the search results, which are responsive to the query, ordered according to scores from the second candidate scoring function; receiving user input selecting one of the first and second candidate scoring functions over the other; and using the user input to inform future selections of pairs of candidate scoring functions from the group of candidate scoring functions. ### Claim 13 (depends on 12) The system of claim 12, wherein the operations comprise: classifying the query; and using the classification to select the group of candidate scoring functions from a plurality of available scoring functions. ### Claim 14 (depends on 12) The system of claim 12, wherein the operations comprise selecting the group of candidate scoring functions from a plurality of available scoring functions based on a respective score associated with each scoring function. ### Claim 15 (depends on 12) The system of claim 12, wherein using the user input to inform future selections of pairs of candidate scoring functions from the group of candidate scoring functions comprises: generating a ranking for the candidate scoring functions in the group based at least in part on the user input; and selecting, for evaluation by one or more other users, one or more candidate scoring functions in the group based at least in part on the ranking for the candidate scoring functions in the group. ### Claim 16 (depends on 15) The system of claim 15, wherein the operations comprise: receiving a second query; selecting a third candidate scoring function and a fourth candidate scoring function from a top number of scoring functions in the ranking for the candidate scoring functions in the group; presenting search results, which are responsive to the second query, ordered according to scores from the third candidate scoring function; presenting the search results, which are responsive to the second query, ordered according to scores from the fourth candidate scoring function; receiving second user input selecting one of the third and fourth candidate scoring functions over the other; and updating the ranking for the candidate scoring functions in the group based at least in part on the second user input. ### Claim 17 (depends on 16) The system of claim 16, wherein the operations comprise: applying each respective candidate scoring function of the group to the search results to determine respective scores for the search results for the respective candidate scoring function; and comparing results located in same positions in respective sets of the search results, ordered according to respective candidate scoring functions, for each of the pairs of candidate scoring functions; wherein the selecting is based on output of the comparing. ### Claim 18 (depends on 17) The system of claim 17, wherein the comparing comprises comparing results in first through tenth positions in the respective sets of the search results. ### Claim 19 (depends on 17) The system of claim 17, wherein the applying comprises providing signals including information about the query and each result in the search results to each respective candidate scoring function. ### Claim 20 (depends on 16) The system of claim 16, wherein generating a ranking for the candidate scoring functions in the group based at least in part on the user input comprises: identifying a user and an associated candidate scoring function from the group of candidate scoring functions, the user having provided the user input, and the user being associated with the associated candidate scoring function; and determining a number of inputs received from different users, including the identified user, associated with the associated candidate scoring function; and generating the ranking for the candidate scoring functions in the group based at least in part on the number of inputs received from the different users. ### Claim 21 (depends on 16) The system of claim 16, wherein generating a ranking for the candidate scoring functions in the group based at least in part on the user input comprises: adding the user input to a set of multiple pair-wise user comparisons of scoring functions; and generating the ranking for the candidate scoring functions in the group based at least in part on the set of multiple pair-wise user comparisons. ### Claim 22 (depends on 21) The system of claim 21, wherein the search results ordered according to scores from the first candidate scoring function constitute a first order, and the operations comprise: receiving a notification that the first candidate scoring function has been modified; applying the modified first scoring function to the search results to determine modified scores for the search results and to generate a second order for the search results according to the modified scores; determining that the first order and the second order are different; and modifying the ranking for the candidate scoring functions in the group so that the modified ranking for the candidate scoring functions in the group does not reflect the user input. ### Claim 23 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform actions comprising: receiving a plurality of queries sequentially, each query being received from a respective user; and performing the following operations with respect to each query and respective user: selecting a pair of candidate scoring functions from a group of at least three candidate scoring functions, the pair including a first candidate scoring function and a second candidate scoring function, the selecting being based at least in part on a ranking of the candidate scoring functions in the group; presenting search results, which are responsive to the query, ordered according to scores from the first candidate scoring function, to the respective user; presenting the search results, which are responsive to the query, ordered according to scores from the second candidate scoring function, to the respective user; receiving user input, from the respective user, selecting one of the first and second candidate scoring functions over the other; and using the user input to update the ranking of the candidate scoring functions in the group. ### Claim 24 (depends on 23) The system of claim 23, wherein using the user input to update the ranking of the candidate scoring functions in the group comprises: adding the user input to a set of multiple pair-wise user comparisons of scoring functions; and generating the ranking of the candidate scoring functions in the group based at least in part on the set of multiple pair-wise user comparisons. ### Claim 25 (depends on 23) The system of claim 23, wherein selecting a pair of candidate scoring functions from a group of at least three candidate scoring functions comprises: determining a diversity score between the first candidate scoring function and the second candidate scoring function, wherein the diversity score is based at least in part on a comparison of respective search result sets ordered according to the first candidate scoring function and the second candidate scoring function; and determining that the diversity score between the first scoring function and the second scoring function satisfies a threshold.