Deriving and Using Interaction Profiles

A method performed by a system of one or more computers, the method comprising:

• identifying one or more classes of search queries;
• comparing, by the system, for each class of search queries, a first scoring algorithm for scoring electronic documents responsive to search queries in the class with a second scoring algorithm for scoring the electronic documents responsive to search queries in the class, wherein the comparing includes:
• obtaining a first profile that characterizes historical interaction, by a first plurality of users, with electronic documents in first result sets of electronic documents generated in response to search queries in the class, the electronic documents in each first result set scored by a search engine in accordance with the first scoring algorithm and ranked by the search engine in each first result set in accordance with the scoring,
• obtaining a second profile that characterizes historical interaction, by a second plurality of users, with electronic documents in second result sets of electronic documents generated in response to search queries in the class, the electronic documents in each second result set scored by the search engine in accordance with the second scoring algorithm and ranked by the search engine in each second result set in accordance with the scoring,
• determining that one or more metrics of the user interaction characterized in the first profile collectively indicate a higher level of user satisfaction with the first result sets of electronic documents than a level of user satisfaction with the second result sets indicated by the one or more metrics of the user interaction characterized in the second profile, and
• in response to the determination, outputting an indication that the first scoring algorithm is better than the second scoring algorithm for scoring documents in result sets generated in response to search queries in the class.

A system comprising:

• one or more computers programmed to perform operations, the operations comprising:
• identifying one or more classes of search queries;
• comparing, by the system, for each class of search queries, a first scoring algorithm for scoring electronic documents responsive to search queries in the class with a second scoring algorithm for scoring the electronic documents responsive to search queries in the class, wherein the comparing includes:
• obtaining a first profile that characterizes historical interaction, by a first plurality of users, with electronic documents in first result sets of electronic documents generated in response to search queries in the class, the electronic documents in each first result set scored by a search engine in accordance with the first scoring algorithm and ranked by the search engine in each first result set in accordance with the scoring,
• obtaining a second profile that characterizes historical interaction, by a second plurality of users, with electronic documents in second result sets of electronic documents generated in response to search queries in the class, the electronic documents in each second result set scored by the search engine in accordance with the second scoring algorithm and ranked by the search engine in each second result set in accordance with the scoring,
• determining that one or more metrics of the user interaction characterized in the first profile collectively indicate a higher level of user satisfaction with the first result sets of electronic documents than a level of user satisfaction with the second result sets indicated by the one or more metrics of the user interaction characterized in the second profile, and
• in response to the determination, outputting an indication that the first scoring algorithm is better than the second scoring algorithm for scoring documents in result sets generated in response to search queries in the class.