US 10,477,095 B2Granted

Selecting optimal image from mobile device captures

AssigneeSRI International

InventorsDavid Chao Zhang, John Benjamin Southall, Michael Anthony Isnardi, Michael Raymond Piacentino, David Christopher Berends, Girish Acharya, Douglas A. Bercow, Aaron Spaulding, Sek Chai

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Device logic in a mobile device configures a processor to capture a series of images, such as a video, using a consumer-grade camera, and to analyze the images to determine the best-focused image, of the series of images, that captures a region of interest. The images may be of a textured surface, such as facial skin of a mobile device user. The processor sets a focal length of the camera to a fixed position for collecting the images. The processor may guide the user to position the mobile device for capturing the images, using audible cues. For each image, the processor crops the image to the region of interest, extracts luminance information, and determines one or more energy levels of the luminance via a Laplacian pyramid. The energy levels may be filtered, and then are compared to energy levels of the other images to determine the best-focused image.

Claims15

Dependent claims

A method of capturing, using a mobile device, a best-focused image of a skin surface of a subject, the method comprising:

setting a camera of the mobile device to a fixed focal length;
capturing, using the camera, a current image of a plurality of images of the skin surface, the plurality of images having a sequence and including a first previous image captured, using the camera, previously to the current image and a second previous image captured, using the camera, previously to the first previous image;
producing a modified image from the current image;
transforming the modified image, using a Laplacian pyramid, to produce a plurality of first luminance values from the modified image and a plurality of second luminance values from the plurality of first luminance values;
averaging a plurality of first squared values, each comprising a square of a corresponding first luminance value of the plurality of first luminance values, to produce a first energy value;
averaging a plurality of second squared values, each comprising a square of a corresponding second luminance value of the plurality of second luminance values, to produce a second energy value;
calculating a first ratio of the first energy value to the second energy value;
calculating, as an average first energy value of the first previous image, an average of the first energy value, a corresponding first energy value of the first previous image, and a corresponding first energy value of the second previous image;
calculating, as an average first ratio of the first previous image, an average of the first ratio, a corresponding first ratio of the first previous image, and a corresponding first ratio of the second previous image;
determining that the first previous image is one of a plurality of valid images, wherein each valid image of the plurality of valid images is an image of the plurality of images and has:
a corresponding average first energy value above an energy threshold value;
and a corresponding average first ratio approximately equal to 1.0;
determining that a first valid image of the plurality of valid images is the best-focused image, wherein the first valid image has a corresponding average first energy value that is greater than the corresponding average first energy values of:
a previous valid image captured immediately before the first valid image;
and a subsequent valid image captured immediately after the first valid image;
and
performing an action associated with the best-focused image.

14 more elements

A method of determining, using a mobile device, a best-focused image of a textured surface, the method comprising:

receiving a plurality of images of the textured surface, the plurality of images captured by a camera of the mobile device while the camera is set to a focal length;
applying a Laplacian pyramid to a first image of the plurality of images to generate a Laplacian pyramid having a first level based on the first image and a second level based on the first level;
determining a first energy value of the first image, the first energy value representing a Laplacian energy of the first level;
determining a second energy value of the first image, the second energy value representing a Laplacian energy of the second level;
determining that the first energy value exceeds an energy threshold and is approximately equal to the second energy value;
determining, based at least in part on the first energy value of the first image and a corresponding first energy value of each valid image of a plurality of valid images, that the first image is the best-focused image, the plurality of valid images comprising all of the plurality of images having a corresponding first energy value that exceeds the energy threshold and is approximately equal to a corresponding second energy value;
and
performing an action associated with the best-focused image;
wherein receiving the plurality of images comprises controlling the camera to capture the plurality of images at a capture rate as the mobile device is continuously moved between a first point that is a first distance away from the textured surface and a second point that is a second distance away from the textured surface, the first distance being less than the focal length and the second distance being greater than the focal length;
wherein the plurality of images includes a current image captured immediately subsequently to the first image, and wherein performing the action comprises controlling the camera to stop capturing the plurality of images such that the current image is captured last of the plurality of images;
wherein determining that the first image is the best-focused image comprises:
calculating an average first energy value from corresponding first energy values of the first image, the current image, and a previous image of the plurality of images, the previous image being captured immediately previously to the first image;
calculating an average first ratio from of a plurality of first ratios, each associated with a corresponding image of the plurality of images and comparing a corresponding first energy value of the image to a corresponding second energy value of the image, the plurality of first ratios including a corresponding first ratio for each of the first image, the previous image, and the current image;
determining that the average first energy value exceeds the energy threshold and the average first ratio is approximately 1.0;
determining that the average first energy value of the first image is greater than a corresponding average first energy value of each of:
a previous valid image of the plurality of valid images, the previous valid image captured immediately before the first image;
and
a subsequent valid image of the plurality of valid images, the subsequent valid image captured immediately after the first image.

15 more elements

A mobile device comprising:

a camera set to a focal length;
memory storing device logic and a Laplacian pyramid;
and
a processor in electronic communication with the memory and the camera, the processor executing the device logic to:
receive, from the camera, a plurality of images of a region of interest;
apply the Laplacian pyramid to each image to generate a corresponding Laplacian pyramid having a first level based on the image and a second level based on the first level;
determine, for each image of the plurality of images, a corresponding first energy value representing Laplacian energy of the first level of the corresponding Laplacian pyramid;
determine, for each image of the plurality of images, a corresponding second energy value representing Laplacian energy of the second level of the corresponding Laplacian pyramid;
determine a best-focused image of the plurality of images based at least in part on the corresponding first energy value of each valid image of a plurality of valid images including the best-focused image, wherein each valid image of the plurality of valid images is one of the plurality of images and has a first energy value that exceeds an energy threshold and is approximately equal to a second energy value;
and
perform an action based on the best-focused image;
a range sensor in electronic communication with the processor, wherein the region of interest includes a textured surface and to receive the plurality of images, the processor executes the device logic to:
receive a signal from the range sensor;
determine that the signal indicates that the range sensor detected that the mobile device is positioned at a first point that is a first distance away from the textured surface, the first distance being less than the focal length;
and
control the camera to capture the plurality of images at a capture rate as the mobile device is continuously moved from the first point to a second point that is a second distance away from the textured surface, the second distance being greater than the focal length;
wherein the processor receives each image of the plurality of images as the image is captured, such that the plurality of images has a first sequence and the plurality of valid images has a second sequence comprising the first sequence with each image of the plurality of images that is not one of the plurality of valid images removed;
wherein each valid image of the plurality of valid images further has a corresponding average first energy value and a corresponding average second energy value that exceed the energy threshold and are approximately equal, the processor executing the device logic to:
calculate the corresponding average first energy value from the corresponding first energy values of the valid image, a previous image positioned immediately before the valid image in the first sequence, and a subsequent image positioned immediately after the valid image in the first sequence;
calculate the corresponding average second energy value from the corresponding second energy values of the valid image, the previous image, and the subsequent image;
and
determine that the corresponding average first energy value exceeds the energy threshold and is approximately equal to the corresponding average second energy value;
wherein to determine the best-focused image, the processor executes the device logic to:
determine that the corresponding average first energy value of a first image of the plurality of valid images is greater than the corresponding average first energy value of each of:
a previous valid image of the plurality of valid images, the previous valid image positioned immediately before the first image in the second sequence;
and
a subsequent valid image of the plurality of valid images, the subsequent valid image positioned immediately after the first image in the second sequence;
and
wherein to perform the action, the processor executes the device logic to control the camera to stop capturing the plurality of images.

26 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/573,325Filed 19 Jan 2016 · us provisional application
Priority: 14 May 2015
Published: US 2018/0139377 A117 May 2018
Granted: 12 Nov 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification15 codes

A61B 5/0077
Using light, e.g. diagnosis by transillumination, diascopy, fluorescence: devices for viewing the surface of the body, e.g. camera, magnifying lens
A61B 5/442
Skin evaluation, e.g. for skin disorder diagnosis: evaluating skin mechanical properties, e.g. elasticity, hardness, texture, wrinkle assessment
G06T 5/20
Image enhancement or restoration: using local operators
G06T 7/0012
Inspection of images, e.g. flaw detection: biomedical image inspection