Back to results
US 2019/0095465 A1Published application
Object based image search
AssigneePinterest, Inc.
InventorsAndrew Huan Zhai, Zhiyuan Zhang, Kevin Yushi Jing, Dmitry Olegovich Kislyuk
Life of this patent
Estimating the term…
Abstract
Described is a system and method for enabling visual search for information. With each selection of an object included in an image, additional images that include visually similar objects are determined and presented to the user.
Claims20
A computing system, comprising:
- an image data store storing:
- a first plurality of stored images, each of the first plurality of stored images having a plurality of image segments;
- and
- image information corresponding to each image, the image information indicating for each image:
- a respective plurality of image segments, wherein each image segment represents a portion of a respective stored image that is less than the entire stored image;
- and
- a plurality of stored feature vectors, each stored feature vector corresponding to an object represented in an image segment of the plurality of image segments;
- one or more processors;
- and
- a memory storing program instructions that when executed by the one or more processors cause the one or more processors to at least:
- receive from a user device, an image as part of a visual based search;
- process the image to determine an object of interest represented in the image;
- generate an object feature vector representative of the object of interest;
- compare the object feature vector with the plurality of stored feature vectors to determine a second plurality of stored feature vectors that include representations of objects that are visually similar to the object of interest;
- determine, based at least in part on the comparison of the object feature vector with the plurality of stored feature vectors, a ranked list indicating a second plurality of images of the first plurality of stored images, each of the second plurality of images including at least one image segment that includes a representation of an object that is determined to be visually similar to the object of interest based at least in part on the comparison;
- and
- send for presentation on a display of the user device, each image of the second plurality of images, such that the presentation includes the entire image of each image of the second plurality of images.
14 more elements
A computer-implemented method, comprising:
- receiving, from a user device, an indication of an image;
- processing the image to determine a first object represented in the image;
- generating an object feature vector representative of the first object;
- comparing the object feature vector with a plurality of stored feature vectors, each of the plurality of stored feature vectors representative of a respective image segment of an image of a first plurality of images, each image segment representing less than all of the respective image;
- and
- generating a ranked list of a second plurality of images from the first plurality of images, wherein each image of the second plurality of images include at least one respective image segment that is determined, based at least in part on the comparing, to include a representation of an object that is visually similar to the first object;
- and
- sending at least one of the second plurality of images or the ranked list such that each of the second plurality of images are presented by the user device.
5 more elements
A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of a computing system, cause the computing system to at least:
- maintain in a data store image information corresponding to a plurality of images, the image information indicating for each image:
- a respective plurality of image segments, wherein each image segment represents a portion of a respective image;
- a respective plurality of feature vectors, wherein each feature vector representative of an object within a respective image segment;
- and
- a plurality of labels, each label corresponding to a respective image segment;
- determine an object represented in an image;
- generate an object feature vector representative of the object;
- determine a label for the object;
- determine, based at least in part on the label, a sub-plurality of the plurality feature vectors;
- compare the object feature vector with each of the sub-plurality of feature vectors to determine similarity scores, each similarity score representative of a similarity between the object feature vector and a respective feature vector of the sub-plurality of feature vectors;
- and
- generate, based at least in part on the similarity scores, a ranked list of stored images.
9 more elements
Dependencies are inferred from the claim text.