Searching for images by video

A method comprising:

• receiving a first frame of a video clip comprising an object of interest;
• creating a representation of the object of interest in the video clip;
• retrieving images from a database based at least in part on the representation of the object of interest in the first frame of the video clip;
• calculating a first similarity score for each image of the retrieved images based on similarities between the retrieved images from the database and the representation of the first frame to identify candidate search images from the retrieved images;
• receiving a second frame of the video clip comprising the object of interest;
• analyzing the second frame of the video clip comprising the object of interest to identify background items to help identify the object of interest by calculating gradients of functions of the identified candidate search images and images in the video clip to determine temporal consistency of the identified background items surrounding the identified object of interest that should remain constant in frames of the video clip relative to the identified object of interest;
• ranking at least one of the identified, candidate search images based on the respective first similarity score for the at least one identified candidate search image and re-ranking at least one of the identified candidate search images based on the determined temporal consistency of the identified background items surrounding the identified object of interest that should remain constant in frames of the video clip relative to the identified object of interest;
• and
• displaying the re-ranked at least one of the identified candidate search images.

One or more computer-readable storage media storing instructions that, when executed by a processor, perform acts comprising:

• receiving a first frame of a video clip comprising an object of interest;
• creating a representation of the object of interest in the first frame of the video clip;
• retrieving images from a database based at least in part on the representation of the object of interest in the first frame of the video clip;
• calculating a first similarity score for each image of the retrieved images based on similarities between the retrieved images from the database and the representation of the first frame to identify candidate search images from the retrieved images;
• receiving a second frame of the video clip comprising the object of interest;
• analyzing the second frame of the video clip comprising the object of interest to identify background items to help identify the object of interest by calculating gradients of functions of the identified candidate search images and images in the video clip to determine temporal consistency of the identified background items surrounding the identified object of interest that should remain constant in frames of the video clip relative to the identified object of interest;
• ranking at least one of the identified candidate search images based on the respective first similarity score for the at least one identified candidate search image and re-ranking at least one of the identified candidate search images based on the determined temporal consistency of the identified background items surrounding the identified object of interest that should remain constant in frames of the video clip relative to the identified object of interest;
• and
• displaying the re-ranked at least one of the identified candidate search images.