# US 2013/0170743 A1 · Detecting Illumination in Images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0170743 A1
- Document ID: US20130170743A1
- Status: Published application
- Kind: A1
- Application number: 13/764,833
- Filed: 2013-02-12
- Published: 2013-07-04
- Earliest claimed: 2009-12-17
- Assignees: Apple Inc.
- Inventors: Graham D. Finlayson; Mark Samuel Drew; Clement Fredembach
- Web page: https://patentgrep.com/patent/US20130170743A1

## CPC

- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 2207/10024: Color image
- G06T 2207/10152: Varying illumination
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image having m light sources, with m preferably equaling 2 or 3, is segmented into different regions, each of which is lit by only one of the m light sources, by obtaining paired imaged with different filtering, for example a filtered and an unfiltered image, applying to the image pairs sets of m pre-computed mappings at the pixel or region level, and selecting the most appropriate. The rendering of the information in the image maybe adjusted accordingly.

## Claims

### Claim 1

A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component;
obtain a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component;
obtain mappings between regions in the first image and spatially corresponding regions in the second image;
apply the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value's corresponding region is illuminated by 1 of the m light sources;
identify that mapping whose associated value is most indicative of one of the rn light sources; and
identify the one light source s an illuminant of the scene.

### Claim 2 (depends on 1)

The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain first and second images comprise instructions to cause the one or more processors to capture the first and second images with an imaging device.

### Claim 3 (depends on 2)

The non-transitory program storage device of claim 2, wherein the instructions to cause the one or more processors to capture the first and second images comprise instructions to cause the one or more processors to capture the first image with a first sensor and the second image with a second different sensor of the imaging device.

### Claim 4 (depends on 1)

The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain first and second images comprise instructions to cause the one or more processors to retrieve first and second images from a storage device.

### Claim 5 (depends on 1)

The non-transitory program storage device of claim 1, wherein the first image comprises an RGB image and the second image comprises a filtered RGB image.

### Claim 6 (depends on 1)

The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain mappings comprise instructions to cause the one or more processors to obtain N mappings, where N>m.

### Claim 7 (depends on 6)

The non-transitory program storage device of claim 6, wherein the instructions to cause the one or more processors to apply the mappings comprise instructions to cause the one or more processors to apply N-choose-m combinations of the N mappings.

### Claim 8 (depends on 6)

The non-transitory program storage device of claim 6, wherein the instructions to cause the one or more processors to apply the mappings comprise instructions to cause the one or more processors to segment the first image into different regions, each of which is lit by only one of the m light sources.

### Claim 9 (depends on 7)

The non-transitory program storage device of claim 7, wherein the instructions to cause the one or more processors to identify that mapping whose associated value is most indicative of one of the m light sources, comprise instructions to cause the one or more processors to identify that mapping that has the lowest prediction error.

### Claim 10 (depends on 1)

The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to identify that mapping whose associated value is most indicative of one of the m light sources comprise instructions to cause the one or more processors to identify that mapping that, based on the first image, most closely predicts the second image.

### Claim 11 (depends on 1)

The non-transitory program storage device of claim 1, further comprising instructions to cause the one or more processors to render the first image based, at least in part, on the identification of the rn illuminants,

### Claim 12 (depends on 1)

The non-transitory program storage device of claim 1, wherein the instructions to cause the one or more processors to obtain mappings comprise instructions to cause the ne or more processors to apply robust statistical procedures to generate the mappings.

### Claim 13

A method of processing an image, comprising:
obtaining a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component;
obtaining a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component;
obtaining mappings between regions in the first image and spatially corresponding regions in the second image;
applying the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value's corresponding region is illuminated by 1 of the m light sources;
identifying that mapping whose associated value is most indicative of one of the m light sources; and
identifying the one light source as an illuminant of the scene.

### Claim 14 (depends on 13)

The method of claim 13, wherein the act of obtaining first and second images comprises capturing the first and second images with an imaging device.

### Claim 15 (depends on 14)

The method of claim 14, wherein the act of capturing the first and second images comprises capturing the first image with a first sensor and the second image with a second different sensor of the imaging device.

### Claim 16 (depends on 13)

The method of claim 13, wherein the act of obtaining first and second images cot prises retrieving first and second images from a storage device.

### Claim 17 (depends on 13)

The method of claim 13, wherein the first image comprises an RGB image and the second image comprises a filtered RGB image.

### Claim 18 (depends on 13)

The method of claim 13, wherein the act of obtaining mappings comprises obtaining N mappings, where N>m.

### Claim 19 (depends on 18)

The method of claim 18, wherein the act of applying the mappings comprises applying N-choose-rn combinations of the N mappings.

### Claim 20 (depends on 18)

The method of claim 18, wherein the act of applying the mappings comprises segmenting the first image into different regions, each of which is lit by only one of the m light sources.

### Claim 21 (depends on 19)

The method of claim 19, wherein the act of identifying that mapping whose associated value is most indicative of one of the m light sources, comprises identifying that mapping that has the lowest prediction error.

### Claim 22 (depends on 13)

The method of claim 13, wherein the act of identifying that mapping whose associated value is most indicative of one of the m light sources comprises identifying that mapping that, based on the first image, most closely predicts the second image.

### Claim 23 (depends on 13)

The method of claim 13, further comprising rendering the first image based, at least in part, on the identification of the m illuminants.

### Claim 24 (depends on 13)

The method of claim 13, wherein the act of obtaining mappings comprises applying robust statistical procedures to generate the mappings.

### Claim 25

An electronic device, comprising:
memory;
one or more image capture sensors operatively coupled to the memory;
a display operatively coupled to the memory; and
one or more processors operatively coupled to the memory, one or more image capture sensors and the display and configured to execute instructions stored in the memory to cause the one or more processors to—
obtain a first image of a scene that is illuminated with m light sources, wherein the first image is associated with a first spectral component,
obtain a second image of the scene, wherein the second image is associated with a second spectral component that is different from the first spectral component,
obtain mappings between regions in the first image and spatially corresponding regions in the second image,
apply the mappings to regions of the first image to obtain values for the spatially corresponding regions of the second image, wherein each value is indicative of the likelihood that the value's corresponding region is illuminated by 1 of the m light sources,
identify that mapping whose associated value is most indicative of one of the m light sources, and
identify the one light source as an illuminant of the scene.