Semantic image similarity
AssigneeMicrosoft Technology Licensing, LLC
InventorsManish GUPTA, Niraj Nrisinvha BHILEGAONKAR, Avishek MAZUMDER, Akash KATARE
Life of this patent
Estimating the term…
Abstract
In various examples, an image search is performed based on a query image where the result includes images with semantic information that matches the query image. For example, image encoders are used to extract information from images that are stored in an index. Continuing this example, when a query is obtained the index is searched to determine potential matches, and a comparison is performed between the query image and the potential matches to determine images with semantic information matching the query image.
Claims20
A computer-implemented method, comprising:
- receiving a query image;
- extracting, using one or more machine-learned models, semantic structure information from the query image, the semantic structure information representing relationships among elements depicted in the query image;
- accessing an index storing semantic structure information corresponding to a plurality of stored visual representations;
- determining, based on a comparison between the semantic structure information extracted from the query image and the semantic structure information stored in the index, one or more stored visual representations that are semantically similar to the query image;
- and
- providing an output identifying the one or more stored visual representations.
A method, comprising:
- receiving a query image;
- identifying a plurality of elements depicted in the query image;
- generating, based on the plurality of elements, structured relationship information indicating relationships among the plurality of elements;
- searching an index of stored images using the structured relationship information;
- identifying one or more stored images having structured relationship information that corresponds to the structured relationship information of the query image;
- and
- providing a result based on the one or more stored images.
A system, comprising:
- a processor;
- and
- memory storing instructions that, as a result of being executed by the processor, cause the processor to:
- obtain training data including a set of labeled pairs of images, labeled pairs of the set of labeled pairs of images indicating whether images contain similar semantic information;
- train a plurality of image encoders, based on the training data, to generate embeddings representing semantic information depicted in input images;
- generate an index including embeddings generated by the plurality of image encoders based on a set of images;
- receive a query image;
- generate, using the plurality of image encoders, a query embedding based on the query image;
- search the index based on the query embedding to identify one or more stored images associated with embeddings that are similar to the query embedding;
- and
- provide an output based on the one or more stored images.
Dependencies are inferred from the claim text.