# US 10,375,317 B2 · Low complexity auto-exposure control for computer vision and imaging systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,317 B2
- Document ID: US10375317B2
- Status: Granted
- Kind: B2
- Application number: 15/642,303
- Filed: 2017-07-05
- Published: 2018-02-15
- Granted: 2019-08-06
- Earliest claimed: 2016-07-07
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Victor Chan
- Web page: https://patentgrep.com/patent/US10375317B2

## CPC

- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/24: Classification techniques
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space
- H04N 23/73: by influencing the exposure time
- H04N 23/76: by influencing the image signals
- G06V 10/467: {Encoded features or binary features, e.g. local binary patterns [LBP]}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods, apparatuses, computer-readable medium, and systems are disclosed for performing automatic exposure control (AEC). In one embodiment, a first digital image is captured while applying a first set of values to one or more exposure control parameters. At least one computer vision (CV) operation is performed using image data from the first digital image, thereby generating a first set of CV features from a set of possible CV features. A mask is obtained comprising a value for each feature of the set of possible CV features. Using the mask, a first measure of abundance is obtained of relevant CV features among the first set of CV features extracted from the first digital image. Based on the first measure of abundance of relevant CV features, an updated set of values is generated for applying to the one or more exposure control parameters for capturing a subsequent digital image of the scene.

## Claims

### Claim 1

A method for performing automatic exposure control (AEC) comprising:
capturing a first digital image of a scene using an image sensor while applying a first set of values to one or more exposure control parameters;
performing object detection on the first digital image, wherein the object detection identifies one or more objects in the first digital image, wherein the object detection outputs a first set of features extracted from the first digital image, wherein a feature is a numerical value indicating gradients between a set of neighboring pixels, and wherein the first set of features includes a count of occurrences in the first digital image of each feature of a set of possible features;
obtaining a mask associated with one or more objects in the first digital image, the mask having a zero value or a non-zero value for each feature of the set of possible features, wherein the mask has a non-zero value for each feature of the set of possible features that is used by the object detection to classify contents of an image as the one or more objects;
applying the mask to the first set of features to determine a subset of the first set of features, the subset of the first set of features including each feature from the first set of features that corresponds to a non-zero value in the mask;
computing a first score using the subset of the first set of features; and
generating, based on the first score, an updated set of values for applying to the one or more exposure control parameters for capturing a subsequent digital image of the scene.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
capturing a second digital image of the scene using the image sensor while applying a second set of values to the one or more exposure control parameters;
performing object detection on the second digital image, wherein the object detection identifies the one or more objects in the second digital image, wherein the object detection outputs a second set of features extracted from the second digital image, the second set of features including a second count of occurrences in the second digital image of each feature of the set of possible features;
applying the mask to the second set of features to determine a subset of the second set of features;
computing a second score using the subset of the second set of features; and
comparing the first score to the second score, wherein generating the updated set of values further includes selecting the second set of values as the updated set of values based on the second score being greater than the first score.

### Claim 3 (depends on 1)

The method of claim 1, wherein the first set of values corresponds to a nominal level of exposure, and further comprising:
capturing a second digital image of the scene using the image sensor while applying a second set of values to the one or more exposure control parameters, wherein the second set of values correspond to a level of exposure greater than the nominal level;
capturing a third digital image of the scene using the image sensor while applying a third set of values to the one or more exposure control parameters, wherein the third set of values correspond to a level of exposure less than the nominal level;
performing object detection on the second digital image, wherein the object detection identifies the one or more objects in the second digital image, wherein the object detection outputs a second set of features extracted from the second digital image, the second set of features including a second count of occurrences in the second digital image of each feature of the set of possible features;
applying the mask to the second set of features to determine a subset of the second set of features;
determining a second score using the subset of the second set of features;
performing object detection on the third digital image, wherein the object detection identifies the one or more objects in the third digital image, wherein the object detection outputs a third set of features extracted from the third digital image, the third set of features including a third count of occurrences in the third digital image of each feature from of the set of possible features;
applying the mask to the third set of features to determine a subset of the third set of features;
determining a third score using the subset of the third set of features; and
comparing the first score, the second score, and the third score to determine a largest score wherein generating the updated set of values further includes selecting, as the updated set of values, one of the first set of values, second set of values, or third set of values corresponding to the largest score.

### Claim 4 (depends on 1)

The method of claim 1, wherein the mask comprises binary values.

### Claim 5 (depends on 1)

The method of claim 1, wherein the mask comprises real values representing weights.

### Claim 6 (depends on 1)

The method of claim 1, further comprising
capturing a digital image of the scene using the image sensor while applying an alternate set of values to the one or more exposure control parameters;
generating a histogram of pixel brightness using image data from the digital image;
determining a desired exposure adjustment based on the histogram of pixel brightness;
generating an updated alternate set of values for applying to the one or more exposure control parameters; and
capturing a next digital image of the scene until a target condition is met, wherein each next digital image of the scene is captured using a different alternate set of values updated according to a respective histogram of pixel brightness for the next digital image.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first digital image is captured subsequent to the target condition being met.

### Claim 8 (depends on 6)

The method of claim 6, wherein the histogram of pixel brightness comprises a first bin associated with a first range of brightness values and a second bin associated with a second range of brightness values, and wherein the desired exposure adjustment is determined based on a first ratio between a dark-bin value representing a pixel count of the first bin and a bright-bin value representing a pixel count of the second bin.

### Claim 9 (depends on 8)

The method of claim 8, wherein the histogram of pixel brightness further comprises a third bin associated with a third range of brightness values and a fourth bin associated with a fourth range of brightness values, and wherein the desired exposure adjustment is further based on a second ratio between the dark-bin value and a black-bin value representing a pixel count of the third bin and the bright-bin value and a white-bin value representing a pixel count of the fourth bin.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
comparing a first value including the dark-bin value and the bright-bin value and a second value including the black-bin value and the white-bin value;
selecting the first ratio or the second ratio based on a result of comparing the first value and the second value; and
using the selected ratio to determine the desired exposure adjustment.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
capturing a next digital image of the scene until a score determined for the next digital image is greater than a threshold, wherein the next digital image is captured while applying a last set of values to the one or more exposure control parameters determines using a previous digital image.

### Claim 12 (depends on 4)

The method of claim 4, wherein determining the first score includes summing each count for each feature in the subset of the first set of features.

### Claim 13 (depends on 5)

The method of claim 5, wherein determining the first score includes summing results of multiplying each count for each feature in the subset of the first set of features by a corresponding weight from the mask.

### Claim 14

An apparatus for performing automatic exposure control (AEC) comprising:
an image sensor for capturing a first digital image of a scene;
at least one processor or circuitry coupled to the image sensor and configured to;
apply a first set of values to one or more exposure control parameters while the first digital image is captured;
perform object detection on the first digital image, wherein the object detection identifies one or more objects in the first digital image, wherein the object detection outputs a first set of features extracted from the first digital image, wherein a feature is a numerical value indicating gradients between a set of neighboring pixels, and wherein the first set of features includes a count of occurrences in the first digital image of each feature of a set of possible features;
obtain a mask associated with one or more objects in the first digital image, the mask having a zero value or a non-zero value for each feature of the set of possible features, wherein the mask has a non-zero value for each feature of the set of possible features that is used by the object detection to classify contents of an image as the one or more objects;
apply the mask to the first set of features to determine a subset of the first set of features, the subset of the first set of features including each feature from the first set of features that corresponds to a non-zero value in the mask;
compute a first score using the subset of the first set of features; and
generate, based on the first score, an updated set of values for applying to the one or more exposure control parameters for capturing a subsequent digital image of the scene.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the at least one processor or circuitry is further configured to:
capture a second digital image of the scene using the image sensor while applying a second set of values to the one or more exposure control parameters;
perform object detection on the second digital image, wherein the object detection identifies the one or more objects in the second digital image, wherein the object detection outputs a second set of features extracted from the second digital image, the second set of features including a second count of occurrences in the second digital image of each feature of set of possible features;
apply the mask to the second set of features to determine a subset of the second set of features;
compute a second score using the subset of the second set of features; and
compare the first score to the second score, wherein generating the updated set of values further includes selecting the second set of values as the updated set of values based on the second score being greater than the first score.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the first set of values corresponds to a nominal level of exposure, and wherein the at least one processor or circuitry is further configured to:
capture a second digital image of the scene using the image sensor while applying a second set of values to the one or more exposure control parameters, wherein the second set of values correspond to a level of exposure greater than the nominal level;
capture a third digital image of the scene using the image sensor while applying a third set of values to the one or more exposure control parameters, wherein the third set of values correspond to a level of exposure less than the nominal level;
perform object detection on the second digital image, wherein the object detection identifies the one or more objects in the second digital image, wherein the object detection outputs a second set of features extracted from the second digital image, the second set of features including a second count of occurrences in the second digital image of each feature of the set of possible features;
apply the mask to the second set of features to determine a subset of the second set of features;
determine a second score using the subset of the second set of features;
perform object detection on the third digital image, wherein the object detection identifies the one or more objects in the third digital image, wherein the object detection outputs a third set of features extracted from the third digital image, the third set of features including a third count of occurrences in the third digital image of each feature from of the set of possible features;
apply the mask to the third set of features to determine a subset of the third set of features;
determine a third score using the subset of the third set of features; and
compare the first score, the second score, and the third score to determine a largest score wherein generating the updated set of values further includes selecting, as the updated set of values, one of the first set of values, second set of values, or third set of values corresponding to the largest score.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the mask comprises binary values.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the mask comprises real values representing weights.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the at least one processor or circuitry is further configured to:
capture a digital image of the scene using the image sensor while applying an alternate set of values to the one or more exposure control parameters;
generate a histogram of pixel brightness using image data from the digital image;
determine a desired exposure adjustment based on the histogram of pixel brightness;
generate an updated alternate set of values for applying to the one or more exposure control parameters; and
capture next digital image of the scene until a target condition is met, wherein each next digital image of the scene is captured using a different alternate set of values updated according to a respective histogram of pixel brightness for the next digital image.

### Claim 20 (depends on 19)

The apparatus of claim 19, the first digital image is captured subsequent to the target condition being met.

### Claim 21 (depends on 19)

The apparatus of claim 19, wherein the histogram of pixel brightness comprises a first bin associated with a first range of brightness values and a second bin associated with a second range of brightness values, and wherein the at least one processor or circuitry is configured determine the desired exposure adjustment based on a first ratio between a dark-bin value representing a pixel count of the first bin and a bright-bin value representing a pixel count of the second bin.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the histogram of pixel brightness further comprises a third bin associated with a third range of brightness values and a fourth bin associated with a fourth range of brightness values, and wherein the desired exposure adjustment is further based on a second ratio between the dark-bin value and a black-bin value representing a pixel count of the third bin and the bright-bin value and a white-bin value representing a pixel count of the fourth bin.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the at least one processor or circuitry is configured to:
compare a first value including the dark-bin value and the bright-bin value and a second value including the black-bin value and the white-bin value;
select the first ratio or the second ratio based on a result of comparing the first value to the second value; and
use the selected ratio to determine the desired exposure adjustment.

### Claim 24 (depends on 14)

The apparatus of claim 14, wherein the at least one processor or circuitry is further configured to:
capture a next digital image of the scene until a score determined for the next digital image is greater than a threshold, wherein the next digital image is captured while applying a last set of values to the one or more exposure control parameters determines using a previous digital image.

### Claim 25 (depends on 17)

The apparatus of claim 17, wherein determining the first score includes summing each count for each feature in the subset of the first set of features.

### Claim 26 (depends on 18)

The apparatus of claim 18, wherein determining the first score includes summing results of multiplying each count for each feature in the subset of the first set of features by a corresponding weight from the mask.

### Claim 27

A non-transitory computer readable medium having instructions stored thereon that, when carried out by one or more processors, causes the one or more processors to:
capture a first digital image of a scene using an image sensor while applying a first set of values to one or more exposure control parameters;
perform object detection on the first digital image, wherein the object detection identifies one or more objects in the first digital image, wherein the object detection outputs a first set of features extracted from the first digital image, wherein a feature is a numerical value indicating gradients between a set of neighboring pixels, and wherein the first set of features includes a count of occurrences in the first digital image of each feature of a set of possible features;
obtain a mask associated with one or more objects in the first digital image, the mask having a zero value or a non-zero value for each feature of the set of possible features, wherein the mask has a non-zero value for each feature of the set of possible features that is used by the object detection to classify contents of an image as the one or more objects;
apply the mask to the first set of features to determine a subset of the first set of features, the subset of the first set of features including each feature from the first set of features that corresponds to a non-zero value in the mask;
compute a first score using the subset of the first set of features; and
generate, based on the first score, an updated set of values for applying to the one or more exposure control parameters for capturing a subsequent digital image of the scene.

### Claim 28 (depends on 27)

The non-transitory computer readable medium of claim 27, wherein the first set of values corresponds to a nominal level of exposure, and wherein the instructions, when carried out by the one or more processors, further cause the one or more processors to:
capture a second digital image of the scene using the image sensor while applying a second set of values to the one or more exposure control parameters, wherein the second set of values correspond to a level of exposure greater than the nominal level;
capture a third digital image of the scene using the image sensor while applying a third set of values to the one or more exposure control parameters, wherein the third set of values correspond to a level of exposure less than the nominal level;
perform object detection on the second digital image, wherein the object detection identifies the one or more objects in the second digital image, wherein the object detection outputs a second set of features extracted from the second digital image, the second set of features including a second count of occurrences in the second digital image of each feature of the set of possible features;
apply the mask to the second set of features to determine a subset of the second set of features;
determine a second score using the subset of the second set of features;
perform object detection on the third digital image, wherein the object detection identifies the one or more objects in the third digital image, wherein the object detection outputs a third set of features extracted from the third digital image, the third set of features including a third count of occurrences in the third digital image of each feature from of the set of possible features;
apply the mask to the third set of features to determine a subset of the third set of features;
determine a third score using the subset of the third set of features; and
compare the first score, the second score, and the third score to determine a largest score wherein generating the updated set of values further includes selecting, as the updated set of values, one of the first set of values, second set of values, or third set of values corresponding to the largest score.

### Claim 29 (depends on 27)

The non-transitory computer readable medium of claim 27, wherein the instructions, when carried out by the one or more processors, further cause the one or more processors to:
capture a digital image of the scene using the image sensor while applying an alternate set of values to the one or more exposure control parameters;
generate a histogram of pixel brightness using image data from the digital image;
determine a desired exposure adjustment based on the histogram of pixel brightness;
generate an updated alternate set of values for applying to the one or more exposure control parameters; and
capture next digital image of the scene until a target condition is met, wherein each next digital image of the scene is captured using a different alternate set of values updated according to a respective histogram of pixel brightness for the next digital image.

### Claim 30

A system for performing automatic exposure control (AEC) comprising:
means for capturing a first digital image of a scene using an image sensor while applying a first set of values to one or more exposure control parameters;
means for performing object detection on the first digital image, wherein the object detection identifies one or more objects in the first digital image, wherein the object detection outputs a first set of features extracted from the first digital image, wherein a feature is a numerical value indicating gradients between a set of neighboring pixels, and wherein the first set of features includes a count of occurrences in the first digital image of each feature of a set of possible features;
means for obtaining a mask associated with one or more objects in the first digital image, the mask having a zero value or a non-zero value for each feature of the set of possible features, wherein the mask has a non-zero value for each feature of the set of possible features that is used by the object detection to classify contents of an image as the one or more objects;
means for applying the mask to the first set of features to determine a subset of the first set of features, the subset of the first set of features including each feature from the first set of features that corresponds to a non-zero value in the mask;
means for computing a first score using the subset of the first set of features; and
means for generating, based on the first score, an updated set of values for applying to the one or more exposure control parameters for capturing a subsequent digital image of the scene.