Training scoring models optimized for highly-ranked results

A computer-implemented method, comprising:

• storing data in a computer system, the data identifying a plurality of training images for a query, wherein each of the training images is classified as being in a positive group of images for the query or a negative group of images for the query according to a respective query-specific preference measure for the image;
• selecting, with the computer system, a first image from either the positive group of images or the negative group of images, and applying a scoring model to the first image to determine a score for the first image;
• selecting, with the computer system, a plurality of candidate images from the other group of images;
• applying, with the computer system, the scoring model to each of the candidate images to determine a respective score for each candidate image, and then selecting a second image from the candidate images, the second image having a highest score;
• and
• determining, with the computer system, that the scores for the first image and the second image fail to satisfy a criterion, wherein the criterion requires that a result of the score of the image selected from the positive group of images minus the score of the image selected from the negative group of images exceeds a threshold, updating the scoring model, and storing the updated scoring model.

A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

• storing data identifying a plurality of training images for a query, wherein each of the training images is classified as being in a positive group of images for the query or a negative group of images for the query according to a respective query-specific preference measure for the image;
• selecting a first image from either the positive group of images or the negative group of images, and applying a scoring model to the first image to determine a score for the first image;
• selecting a plurality of candidate images from the other group of images;
• applying the scoring model to each of the candidate images to determine a respective score for each candidate image, and then selecting a second image from the candidate images, the second image having a highest score;
• and
• determining that the scores for the first image and the second image fail to satisfy a criterion, wherein the criterion requires that a result of the score of the image selected from the positive group of images minus the score of the image selected from the negative group of images exceeds a threshold, updating the scoring model, and storing the updated scoring model.