# US 9,519,839 B2 · Illumination estimation using natural scene statistics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,839 B2
- Document ID: US9519839B2
- Status: Granted
- Kind: B2
- Application number: 14/188,670
- Filed: 2014-02-24
- Published: 2014-08-28
- Granted: 2016-12-13
- Earliest claimed: 2013-02-25
- Assignees: Texas Instruments Incorporated
- Applicants: Texas Instruments Incorporated
- Inventors: Buyue Zhang; Aziz Umit Batur
- Web page: https://patentgrep.com/patent/US9519839B2

## CPC

- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 5/92: based on global image properties
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for estimating illumination of an image captured by a digital system is provided that includes computing a feature vector for the image, identifying at least one best reference illumination class for the image from a plurality of predetermined reference illumination classes using the feature vector, an illumination classifier, and predetermined classification parameters corresponding to each reference illumination class, and computing information for further processing of the image based on the at least one best reference illumination class, wherein the information is at least one selected from a group consisting of color temperature and white balance gains.

## Claims

### Claim 1

A method for estimating illumination of an image captured by a digital system, the method comprising:
computing, with the digital system, ground truth white balance gains for each training image of a plurality of training images;
clustering, with the digital system, the training images into reference illumination classes based on the ground truth white balance gains;
deriving, with the digital system, feature vectors for each training image in each reference illumination class;
training, with the digital system, an illumination classifier based on the feature vectors, wherein classification parameters for each reference illumination class are determined;
computing, with the digital system, a feature vector for the image;
identifying, with the digital system, at least one best reference illumination class for the image from the reference illumination classes using the feature vector for the image, the illumination classifier, and the classification parameters corresponding to each reference illumination class;
computing, with the digital system, information for further processing of the image based on the at least one best reference illumination class, wherein the information is at least one selected from a group consisting of color temperature and white balance gains; and
performing, with the digital system, further processing on the image based on the information.

### Claim 2 (depends on 1)

The method of claim 1, wherein the illumination classifier is a multivariate Gaussian classifier (MVG), and wherein identifying at least one best reference illumination class comprises determining probabilities of observing the image in each of the reference illumination classes.

### Claim 3 (depends on 1)

The method of claim 1, wherein the illumination classifier is a multivariate Gaussian classifier (MVG), and wherein identifying at least one best reference illumination class comprises computing a Mahalanobis distance of the image to each of the plurality of reference illumination classes.

### Claim 4 (depends on 1)

The method of claim 1, wherein the at least one best reference illumination class comprises multiple best reference illumination classes, and wherein computing information for further processing comprises determining the color temperature for the image based on a weighted average of predetermined color temperatures of the multiple best reference illumination classes.

### Claim 5 (depends on 1)

The method of claim 1, wherein the at least one best reference illumination class comprises multiple best reference illumination classes, and wherein computing information for further processing comprises determining the white balance gains for the image as weighted averages of predetermined white balance gains of the multiple best reference illumination classes.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of training images comprises training images captured under multiple lighting conditions in multiple geographical locations, and wherein each training image has a corresponding reference image configured to provide ground truth color temperature and ground truth white balance gains for the training image.

### Claim 7

A computer-implemented method for training an illumination classifier, the method comprising:
computing, with one or more processors, ground truth white balance gains for each training image of a plurality of training images;
clustering, with the one or more processors, the training images into reference illumination classes based on the ground truth white balance gains;
deriving, with the one or more processors, feature vectors for each training image in each reference illumination class;
training, with the one or more processors, an illumination classifier for the reference illumination classes based on the feature vectors, wherein classification parameters for each reference illumination class are determined; and
performing illumination estimation on an image based on the illumination classifier.

### Claim 8 (depends on 7)

The method of claim 7, wherein the plurality of training images comprises training images captured under multiple lighting conditions in multiple geographical locations, and wherein each training image has a corresponding reference image configured to provide ground truth color temperature and the ground truth white balance gains for the training image.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
computing a color temperature for each reference illumination class as a mean of the ground truth color temperatures of reference images corresponding to the training images in the reference illumination class; and
computing white balance gains for each reference illumination class as means of the ground truth white balance gains computed for each training image in the reference illumination class.

### Claim 10 (depends on 7)

The method of claim 7, wherein the illumination classifier is a multivariate Gaussian classifier (MVG) and the classification parameters of a reference illumination class comprise a mean vector and a covariance matrix of the feature vectors of the training images in the reference illumination class.

### Claim 11 (depends on 10)

The method of claim 10, wherein training an illumination classifier includes generating a feature matrix for each reference illumination class by arranging the feature vectors of the training images in the reference illumination class as columns of the feature matrix.

### Claim 12 (depends on 7)

The method of claim 7, wherein deriving feature vectors comprises computing, for each training image, a two-dimensional (2D) chromaticity histogram, reducing influence of dominant object color in the 2D chromaticity histogram, and transforming the 2D chromaticity histogram into a one-dimensional (1D) vector.

### Claim 13

An apparatus configured to estimate illumination of an image, the apparatus comprising:
means for computing ground truth white balance gains for each training image of a plurality of training images;
means for clustering the training images into reference illumination classes based on the ground truth white balance gains;
means for deriving feature vectors for each training image in each reference illumination class;
means for training an illumination classifier based on the feature vectors, wherein classification parameters for each reference illumination class are determined;
means for capturing the image;
means for computing a feature vector for the image;
means for identifying at least one best reference illumination class for the image from the reference illumination classes using the feature vector for the image, the illumination classifier, and the classification parameters corresponding to each reference illumination class;
means for computing information for further processing of the image based on the at least one best reference illumination class, wherein the information is at least one selected from a group consisting of color temperature and white balance gains; and
means for performing further processing on the image based on the information.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the illumination classifier is a multivariate Gaussian classifier (MVG), and wherein the means for identifying at least one best reference illumination class determines probabilities of observing the image in each of the plurality of reference illumination classes.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the illumination classifier is a multivariate Gaussian classifier (MVG), and wherein the means for identifying at least one best reference illumination class computes a Mahalanobis distance of the image to each of the plurality of reference illumination classes.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the at least one best reference illumination class comprises multiple best reference illumination classes, and wherein the means for computing information for further processing determines the color temperature for the image based on a weighted average of predetermined color temperatures of the multiple best reference illumination classes.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein the at least one best reference illumination class comprises multiple best reference illumination classes, and wherein the means for computing information for further processing determines the white balance gains for the image as weighted averages of predetermined white balance gains of the multiple best reference illumination classes.

### Claim 18 (depends on 13)

The apparatus of claim 13, wherein the plurality of training images comprises training images captured under multiple lighting conditions in multiple geographical locations, and wherein each training image has a corresponding reference image configured to provide ground truth color temperature and ground truth white balance gains for the training image.