Back to results
US 9,754,177 B2Granted
Identifying objects within an image
AssigneeMicrosoft Technology Licensing, LLC
InventorsCe Liu, Yair Weiss, Antonio Torralba Barriuso
Life of this patent
Estimating the term…
Abstract
One or more aspects of the subject disclosure are directed towards identifying objects within an image via image searching/matching. In one aspect, an image is processed into bounding boxes, with the bounding boxes further processed to each surround a possible object. A sub-image of pixels corresponding to the bounding box is featurized for matching with tagged database images. The information (tags) associated with any matched images is processed to identify/categorize the sub-image and thus the object corresponding thereto.
Claims20
A computer-implemented method for processing an image, the method comprising:
- creating a first bounding box around a first region of interest in an image;
- creating a second bounding box around a second region of interest in the image;
- creating a third bounding box around the first region of interest in the image;
- determining that the first bounding box and the third bounding box meet a variance threshold;
- determining that the second bounding box does not meet the variance threshold;
- determining the first bounding box and the third bounding box overlap;
- based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;
- extracting a sub-image corresponding to the fourth bounding box;
- querying a remote image-to-image search engine using the sub-image to find a plurality of matched images for the sub-image, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
- determining a first matching value based on matching the sub-image with the first matched image;
- determining that the first matching value is greater than a matching value threshold;
- based on determining that the first matching value is greater than the matching value threshold, labeling the sub-image with a sub-image tag that corresponds to an identity of the second object;
- and
- outputting the image with the labeled sub-image.
11 more elements
A system comprising:
- a memory including an input image;
- and
- a processor configured to:
- create a first bounding box around a first region of interest in the input image;
- create a second bounding box around a second region of interest in the input image;
- create a third bounding box around the first region of interest in the input image;
- determine that the first bounding box and the third bounding box meet a variance threshold;
- determine that the second bounding box does not meet the variance threshold;
- determine the first bounding box and the third bounding box overlap;
- based on the determination that the first bounding box and the third bounding box overlap, merge the first bounding box and the third bounding box into a single fourth bounding box;
- select a sub-image corresponding to the fourth bounding box;
- query an image-to-image search engine using the sub-image to obtain identifying information of a plurality of matched images, the plurality of matched images comprising a first matched image, the first matched image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
- process the identifying information, the process comprising:
- determining a first matching value based on matching the sub-image with the first matched image;
- determining that the first matching value is greater than a matching value threshold;
- and
- based on determining that the first matching value is greater than the matching value threshold, generating an identifier for the sub-image, the identifier corresponding to the category of the second object.
14 more elements
One or more computer-readable hardware storage devices storing computer executable instructions, which upon execution perform operations, comprising:
- inputting an image;
- creating a first bounding box around a first region of interest in the image;
- creating a second bounding box around a second region of interest in the image;
- creating a third bounding box around the first region of interest in the image;
- determining that the first bounding box and the third bounding box meet a variance threshold;
- determining that the second bounding box does not meet the variance threshold;
- determining the first bounding box and the third bounding box overlap;
- based on the determination that the first bounding box and the third bounding box overlap, merging the first bounding box and the third bounding box into a single fourth bounding box;
- extracting a sub-image corresponding to the fourth bounding box;
- determining sub-image features of the sub-image;
- using the sub-image, querying an image-to-image search engine including a plurality of tagged images with features corresponding to the sub-image features of the sub-image, the plurality of tagged images comprising a first tagged image, the first tagged image comprising a second object that corresponds to a first object, the second object having a first identifier that provides a category for the second object;
- determining a first matching value based on matching the sub-image with the first tagged image;
- determining that the first matching value is greater than a matching value threshold;
- and
- based on determining that the first matching value is greater than the matching value threshold, matching the first object corresponding to the sub-image with the first tagged image.
12 more elements
Dependencies are inferred from the claim text.