Back to results
US 2020/0026907 A1Published application
Object detection based on joint feature extraction
AssigneeMicrosoft Technology Licensing, LLC
InventorsDong Chen, Fang Wen, Gang Hua
Life of this patent
Estimating the term…
Abstract
In implementations of the subject matter described herein, a solution for object detection is proposed. First, a feature(s) is extracted from an image and used to identify a candidate object region in the image. Then another feature(s) is extracted from the identified candidate object region. Based on the features extracted in these two stages, a target object region in the image and a confidence for the target object region are determined. In this way, the features that characterize the image from the whole scale and a local scale are both taken into consideration in object recognition, thereby improving accuracy of the object detection.
Claims20
A computer-implemented method comprising:
- characterizing, by at least one processor, at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
- identifying a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
- identifying a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
- determining whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
- responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
- retaining the first candidate object region and discarding the second candidate region, and
- characterizing at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature and characterizing the image in a local scale, which is at the same scale at the whole scale providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
- responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
- retaining both the first and second candidate object regions, and
- extracting at least one second feature at the same scale as the whole scale from the first and second candidate object regions;
- and
- determining, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
9 more elements
A device comprising:
- a processing unit;
- a memory coupled to the processing unit and storing instructions thereon, the instructions, when executed by the processing unit, causing the device to:
- characterize at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
- identify a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
- identify a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
- determine whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
- responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
- retain the first candidate object region and discard the second candidate region, and
- characterize at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature and characterizing the image in a local scale, which is at the same scale as the whole scale, providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
- responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
- retain both the first and second candidate object regions, and
- extract at least one second feature which is at the same scale as the whole scale from the first and second candidate object regions;
- and
- determine, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
11 more elements
The device of claim 10, wherein when executed by the processing unit, further cause the device to identify the first candidate object region by:
- detecting positions of facial landmarks based on the at least one first feature;
- and
- identifying the candidate face region in the image based on the positions of facial landmarks.
The device of claim 11, wherein the instructions, when executed by the processing unit, further cause the device to identify the candidate face region in the image based on the positions of facial landmarks by:
- mapping the positions of facial landmarks to canonical positions based on a predefined mapping;
- and
- identifying the candidate face region in the image based on the canonical positions.
The device of claim 14, wherein the instructions, when executed by the processing unit, further cause the device to construct the first mask by:
- generating an enlarged patch by increasing the predefined size of the patch;
- and
- constructing the first mask by binarizing values of pixels in the image based on the enlarged patch.
A storage device comprising machine-executable instructions, the instructions, when executed on at least one processor causing the processor to:
- characterize at least one first feature from an image, including by extracting via a first feature extraction stage the at least one first feature and characterizing the image in a whole scale providing context around object regions;
- identify a first candidate object region defined by a first bounding box in the image based on the at least one first feature, wherein the first candidate object region includes a candidate face region;
- identify a second candidate object region defined by a second bounding box in the image based on the at least one first feature;
- determine whether the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box;
- responsive to a determination that the first candidate object region defined by the first bounding box is overlapped by the second candidate object region defined by the second bounding box:
- retain the first candidate object region and discard the second candidate region, and
- characterize at least one second feature from the first candidate object region based on the at least one first feature, including by extracting via a second feature extraction stage the at least one second feature and characterizing the image in a local scale, which is at the same scale as the whole scale providing information from within the first candidate object region, the first feature extraction stage and the second feature extraction stage trained based on a joint optimization objective to improve accuracy of object detection;
- responsive to a determination that the first candidate object region defined by the first bounding box is not overlapped by the second candidate object region defined by the second bounding box:
- retain both the first and second candidate object regions, and
- extract at least one second feature at the same scale as the whole scale from the first and second candidate object regions;
- and
- determine, based on the at least one first feature and the at least one second feature, a target object region in the image and a confidence for the determined target object region.
9 more elements
The storage device of claim 19, wherein the instructions, when executed on the processor further cause the processor to:
- detect positions of facial landmarks based on the at least one first feature;
- and
- identify the candidate face region in the image based on the positions of facial landmarks.
Dependencies are inferred from the claim text.