US 8,478,748 B2 · Directly optimizing evaluation measures in learning to rank

Abstract

The present invention provides methods for improving a ranking model. In one embodiment, a method includes the step of obtaining queries, documents, and document labels. The process then initializes active sets using the document labels, wherein two active sets are established for each query, a perfect active set and an imperfect active set. Then, the process optimizes an empirical loss function by the use of the first and second active set, whereby parameters of the ranking model are modified in accordance to the empirical loss function. The method then updates the active sets with additional ranking data, wherein the updates are configured to work in conjunction with the optimized loss function and modified ranking model. The recalculated active sets provide an indication for ranking the documents in a way that is more consistent with the document metadata.

Claims20

Dependent claims

A method by a computing system including a processor and a memory for improving a ranking model, the method comprising:

obtaining a training set, wherein the training set includes:
training queries:
a document set associated with each training query;
and
document labels, from which at least one perfect and imperfect document set permutations can be inferred for each training query;
initializing active sets to include a perfect document set permutation and an imperfect document set permutation for each training query;
optimizing a ranking model in accordance with an empirical loss function, wherein the empirical loss function utilizes the active sets;
identifying, for each training query, the perfect document set permutation for which the ranking model scores lowest, and the imperfect document set permutation for which the ranking model scores highest;
updating, by a processor, the active sets to include at least one identified document set permutation that was not initially present;
and
using the updated active sets to additionally optimize the ranking model.

8 more elements

A computing system for improving a ranking model, the system comprising:

a memory storing an obtained training set, wherein the obtained training set includes:
training queries;
a document set associated with each training query;
and
document labels, from which at least one perfect and imperfect document set permutations can be inferred for each training query;
a component for initializing active sets to include a perfect document set permutation and an imperfect document set permutation for each training query;
a component for optimizing a ranking model in accordance with an empirical loss function, wherein the empirical loss function utilizes the active sets;
a component for identifying, for each training query, the perfect document set permutation for which the ranking model scores lowest, and the imperfect document set permutation for which the ranking model scores highest;
a component for updating the active sets to include at least one identified document set permutation that was not initially present;
and
a component for using the updated active sets to additionally optimize the ranking model.

8 more elements

A computer-readable device storing code, which when executed, runs a method for improving a ranking model, the method comprising:

obtaining a training set, wherein the training set includes:
training queries;
a document set associated with each training query;
and
document labels, from which at least one perfect and imperfect document set permutations can be inferred for each training query;
initializing active sets to include a perfect document set permutation, and an imperfect document set permutation for each training query;
optimizing a ranking model in accordance with an empirical loss function, wherein the empirical loss function utilizes the active sets;
identifying, for each training query, the perfect document set permutation for which the ranking model scores lowest, and the imperfect document set permutation for which the ranking model scores highest;
updating the active sets to include at least one identified document set permutation that was not initially present;
and
using the updated active sets to additionally optimize the ranking model.

8 more elements

Dependencies are inferred from the claim text.

Directly optimizing evaluation measures in learning to rank

Life of this patent

Directly optimizing evaluation measures in learning to rank

Life of this patent

Abstract

Claims20