Search Engine Performance Evaluation Using a Task-based Assessment Metric
AssigneeMicrosoft Corporation
InventorsYang Song, Zhen Liao, Liwei He
Life of this patent
Estimating the term…
Abstract
The subject disclosure is directed towards evaluating search engine implementation performance using a task-based assessment metric. Logged user activities associated with the search engine implementation are processed into sessions and atomic events corresponding to the user activities within the sessions. The atomic events corresponding to the user activities are classified into tasks based on similarity of the queries within the user activities. After applying the task-based assessment metric to task information associated with the search engine implementation, an evaluation mechanism determines indicia of search engine implementation performance in terms of user behavior.
Claims20
In a computing environment, a method performed at least in part on at least one processor, comprising, applying a task-based assessment metric to a search engine implementation, including, examining user activities associated with the search engine implementation, classifying atomic events corresponding to the user activities into tasks, and evaluating each task in accordance with the task-based assessment metric to determine indicia of search engine implementation performance.
In a computing environment, a system, comprising, an evaluation mechanism configured to compare a first search engine implementation with the second search engine implementation using a task-based assessment metric, wherein the evaluation mechanism is further configured to extract first sets of tasks from a dataset for the first search engine implementation and second sets of tasks from another dataset for the second search engine implementation, to apply the task-based assessment metric to the first sets of tasks and the second sets of tasks to produce a first distribution and a second distribution, respectively, and to determine indicia of a performance difference between the first distribution and the second distribution in terms of user behavior.
One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising:
- classifying atomic events into tasks corresponding to user activities associated with a search engine implementation;
- selecting at least one of a user satisfaction metric and a user engagement metric for assessing a performance of the search engine implementation with respect to the tasks;
- and
- differentiating the performance of the search engine implementation from a benchmark associated with standard search engine implementation performance.
Dependencies are inferred from the claim text.