Method, system, and computer readable medium for identifying result images based on an image query

A system, comprising:

• a data store storing one or more query labels associated with a query image and candidate images, the candidate images having labels that correspond to the one or more query labels;
• and
• an image search subsystem coupled to the data store, the image search subsystem comprising one or more computers configured to:
• identify the candidate images matching the one or more query labels associated with the query image;
• generate relevance scores for the candidate images;
• rank the candidate images based on the relevance scores, each relevance score being a relevance measure of a respective candidate image based on a visual similarity score that represents visual similarities of the respective candidate image to the query image, the visual similarity scores being generated by an image similarity model that has been trained based on user similarity feedback and feature distances between images referenced by the user similarity feedback, the user similarity feedback specifying a reference query image and a first of two training images that was identified, by users, as more visually similar than a second of the two training images to the reference query image, each of the feature distances specifying a visual distance between corresponding image feature scores for the reference query image and the first of the two training images;
• determine that a particular candidate image does not have at least a threshold number of nearest neighbors in common with the query image, a nearest neighbor for the particular candidate image being another image having a visual similarity score that satisfies a nearest neighbor threshold, the nearest neighbor threshold being a value indicative of a pre-specified visual distance from the particular candidate image;
• and
• in response to determining that the particular candidate image does not have at least the threshold number of nearest neighbors in common with the query image, removing the particular candidate image from the list of candidate images.

A non-transitory computer readable storage medium storing thereon a computer program comprising instructions that when executed by a computer system, cause the computer system to perform operations including:

• obtaining one or more query labels associated with a query image;
• selecting candidate images matching the one or more query labels;
• generating visual similarity scores for the candidate images, each of the visual similarity scores representing a visual similarity of a respective candidate image to the query image, the visual similarity scores being generated by an image similarity model that has been trained based on user similarity feedback and feature distances between images referenced by the user similarity feedback, the user similarity feedback specifying a reference query image and a first of two training images that was identified, by users, as more visually similar than a second of the two training images to the reference query image, each of the feature distances indicative of a visual distance between corresponding image feature scores for the reference query image and the first of the two training images;
• generating relevance scores for the candidate images based on the visual similarity scores, each relevance score being a relevance measure of a respective candidate image to the query image;
• generating a ranking of the candidate images;
• selecting a highest ranking subset of the candidate images to be referenced by image search results;
• determining that a particular candidate image in the highest ranking subset of the candidate images does not have at least a threshold number of nearest neighbors in common with the query image, a nearest neighbor for the particular candidate image being another image having a visual similarity score that satisfies a nearest neighbor threshold, the nearest neighbor threshold being a value indicative of a pre-specified visual distance from the particular candidate image;
• and
• in response to determining that the particular candidate image does not have at least the threshold number of nearest neighbors in common with the query image, omitting the particular candidate image from the highest ranking subset of the candidate images that are referenced by the image search results.