Back to results
US 10,896,318 B2Granted
Occlusion detection for facial recognition processes
AssigneeApple Inc.
InventorsThorsten Gernoth, Ian R. Fasel, Touraj Tajbakhsh, Jia-Chi Wu
Life of this patent
Estimating the term…
Abstract
Occlusion of facial features may be detected and assessed in an image captured by a camera on a device. Landmark heat maps may be used to estimate the location of landmarks such as the eyes, mouth, and nose of a user's face in the captured image. An occlusion heat map may also be generated for the captured image. The occlusion heat map may include values representing the amount of occlusion in regions of the face. The estimated locations of the eyes, mouth, and nose may be used in combination with the occlusion heat map to assess occlusion scores for the landmarks. The occlusion scores for the landmarks may be used control one or more operations of the device.
Claims20
A method, comprising:
- capturing an image of a user's face using a camera located on a device, the device comprising a computer processor and a memory, wherein the captured image comprises an image captured while illuminating the user's face with an illuminator located on the device;
- generating, using the computer processor and based on the captured image, one or more landmark maps, each landmark map corresponding to a respective landmark feature of one or more landmark features of the user's face, wherein each of the one or more landmark maps has a lower resolution than the captured image, and wherein each of the one or more landmark maps comprises a plurality of values for regions of the captured image on the one or more landmark maps, each value corresponding to a respective region on the one or more landmark maps and representing a likelihood that the respective landmark feature is in the respective region;
- identifying, using the computer processor, locations of one or more landmark features of the user's face on each of the one or more landmark maps based on the plurality of values for the regions on the one or more landmark maps;
- generating, using the computer processor and based on the captured image, an occlusion map, wherein the occlusion map has the lower resolution than the captured image, and wherein the occlusion map comprises a plurality of occlusion value corresponding to a respective region on the occlusion map and occlusion value corresponding to a respective region on the occlusion map and representing a likelihood of occlusion in the respective region;
- assessing, using the computer processor, an occlusion score for at least one landmark feature on at least one landmark map, wherein the occlusion score is assessed from the occlusion values for the regions on the occlusion map that correspond to the location of the at least one landmark feature based on the at least one landmark map;
- and
- discarding the captured image in response to the assessed occlusion score being above a selected occlusion threshold.
4 more elements
A device, comprising:
- a camera;
- at least one illuminator providing infrared illumination;
- circuitry coupled to the camera and the illuminator, wherein the circuitry is configured to:
- capture an image of a user's face using the camera, wherein the captured image comprises an image captured while illuminating the user's face with the at least one illuminator;
- generate one or more landmark maps based on the captured image, each landmark map corresponding to a respective landmark feature of one or more landmark features of the user's face, wherein each of the one or more landmark maps has a lower resolution than the captured image, and wherein each of the one or more landmark maps comprises a plurality of values for regions of the captured image on the one or more landmark maps, each value corresponding to a respective region on the one or more landmark maps and representing a likelihood that the respective landmark feature is in the respective region;
- identifying locations of one or more landmark features of the user's face on each of the one or more landmark maps based on the plurality of values for the regions on the one or more landmark maps;
- generate an occlusion map, based on the captured image, wherein the occlusion map has the lower resolution than the captured image, and wherein the occlusion map comprises a plurality of occlusion values for the regions of the captured image on the occlusion map, each occlusion value corresponding to a respective region on the occlusion map and representing a likelihood of occlusion in the respective region;
- assess an occlusion score for at least one landmark feature on at least one landmark map, wherein the occlusion score is assessed from the occlusion values for the regions on the occlusion map that correspond to the location of the at least one landmark feature based on the at least one landmark map;
- and
- discard the captured image in response to the assessed occlusion score being above a selected occlusion threshold.
7 more elements
A non-transient computer-readable medium including instructions that, when executed by one or more processors, causes the one or more processors to perform a method, comprising:
- capturing an image of a user's face using a camera located on a device, the device comprising a computer processor and a memory, wherein the captured image comprises an image captured while illuminating the user's face with an illuminator located on the device;
- generating, using the computer processor and based on the captured image, one or more landmark maps, each landmark map corresponding to a respective landmark feature of one or more land mark features of the user's face, wherein each of the one or more landmark maps has a lower resolution than the captured image, and wherein each of the one ore more landmark maps comprises a plurality of values for regions of the captured image on the one or more landmark maps, each value corresponding to a respective region on the one or more landmark maps and representing a likelihood that the respective landmark feature is in the respective region;
- identifying, using the computer processor, locations of one or more landmark features of the user's face on each of the one or more landmark maps based on the plurality of values for the regions on the one or more landmark maps;
- generating, using the computer processor and based on the captured image, an occlusion map, wherein the occlusion map has the lower resolution than the captured image, and wherein the occlusion map comprises a plurality of occlusion value corresponding to a respective region on the occlusion map, each occlusion value corresponding to a respective region on the occlusion map and representing a likelihood of occlusion in the respective region;
- assessing, using the computer processor, an occlusion score for at least one landmark feature on at least one landmark map, wherein the occlusion score is assessed from the occlusion values for the regions on the occlusion map that correspond to the location of the at least one landmark feature based on the at least one landmark map;
- and
- discarding the captured image in response to the assessed occlusion score being above a selected occlusion threshold.
4 more elements
Dependencies are inferred from the claim text.