Composition aware image querying

A system comprising:

• one or more processors;
• a computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to:
• receive user input indicating one or more data points at one or more locations of a search input page from a client device, each of the one or more data points indicating a specific query;
• provide each specific query to a trained computer-operated neural language model;
• obtain a query vector for each specific query of the user input from the trained computer-operated neural language model;
• obtain an indexed grid of image vectors from a data structure for each image of an image collection by
• for each image in the image collection, provide the image to a trained computer-operated convolutional neural network that determines one or more queries that the image probabilistically belongs to;
• generate a saliency map of the image using the trained computer-operated convolutional neural network, the saliency map indicating one or more regions of the image that locate the one or more queries within the image, the saliency map including an image vector for each cell of a plurality of cells in the saliency map;
• modify each image from an arbitrary aspect ratio into a downsized image with a fixed size grid;
• associate each downsized image with image vectors of a corresponding saliency map;
• and
• provide an index to the downsized images with the associated image vectors in a data structure;
• compare each query vector to the indexed grid of image vectors for each image;
• based on the comparison, produce a heat map of each specific query for each image;
• for each image, compute an overall score for the image relative to where the one or more data points were located on the search input page based on the heat map for each specific query and the user input;
• and
• provide a prioritized listing of images from the image collection based on the computed overall score for each image.

A computer-implemented method, comprising:

• providing a plurality of sets of training images to a computer-operated convolutional neural network, wherein the computer-operated convolutional neural network processes the plurality of sets of training images to learn to identify features relating to at least one object class of a plurality of object classes, wherein each of the plurality of sets of training images is associated with one object class of the plurality of object classes;
• generating feature vectors for each training image in the plurality of sets of training images using the computer-operated convolutional neural network;
• clustering the feature vectors into a plurality of clusters;
• receiving user input via an application on a client device to initiate an image search, the user input indicating one or more queries assigned to one or more regions of a search input page;
• generating, in response to the received user input, an image search query from the received user input;
• providing for transmission the image search query over a connection to a server, the server including an image search service that obtains one or more composition-specific images responsive to the image search query based on a cosine similarity between a query vector associated with the image search query and one or more indexed vectors of corresponding images from an image collection;
• and
• receiving a set of search results responsive to the image search query from the server, the set of search results including a prioritized listing of the composition-specific images,
• wherein at least one of the plurality of clusters is associated with the image search query.