# US 2015/0103200 A1 · Heterogeneous mix of sensors and calibration thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0103200 A1
- Document ID: US20150103200A1
- Status: Published application
- Kind: A1
- Application number: 14/065,810
- Filed: 2013-10-29
- Published: 2015-04-16
- Earliest claimed: 2013-10-16
- Assignees: Broadcom Corporation
- Applicants: Broadcom Corporation
- Inventors: Gary Lee Vondran, JR.; Charles Dunlop Macfarlane
- Web page: https://patentgrep.com/patent/US20150103200A1

## CPC

- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/13: Edge detection
- G06T 7/85: {Stereo camera calibration}
- H04N 13/133: Equalising the characteristics of different image components, e.g. their average brightness or colour balance
- H04N 13/246: Calibration of cameras
- H04N 13/25: using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
- H04N 17/002: {for television cameras}
- H04N 23/13: with multiple sensors
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10012: Stereo images
- G06T 2207/10016: Video; Image sequence
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Aspects of calibration of sensors are described. In one embodiment, a characteristic associated with a first or a second sensor is identified. The characteristic may be identified before or after assembly of a device including the first and second sensors. In turn, an operating characteristic of at least one of the first sensor or the second sensor may be adjusted based on the identified characteristic. Further, a first image may be captured with the first sensor and a second image may be captured with the second sensor. An attribute of the second image, for example, may be adjusted to substantially address any difference between the attribute of the second image and a corresponding attribute of the first image using the characteristic for calibration. The adjustments described herein may assist various processing techniques which operate on pairs of images, for example, particularly when a heterogeneous mix of sensors is used.

## Claims

### Claim 1

An image processing method, comprising:
identifying a characteristic for calibration associated with at least one of a first sensor or a second sensor, a pixel density of the second sensor being a fraction of the pixel density of the first sensor;
capturing a first image with the first sensor;
capturing a second image with a second sensor; and
adjusting, with a processing circuit, an attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the first image using the characteristic for calibration.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, before adjusting the attribute of the second image, adapting an operating characteristic of at least one of the first sensor or the second sensor to accommodate for at least one of noise, defective pixels, dark current, vignetting, demosaicing, or white balancing using the characteristic for calibration.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
before adjusting the attribute of the second image, downscaling the first image to a downscaled first image, wherein:
adjusting the attribute of the second image comprises adjusting the attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the downscaled first image using the characteristic for calibration.

### Claim 4 (depends on 1)

The method of claim 1, wherein adjusting the attribute of the second image comprises aligning the second image with the first image to substantially address a difference in alignment between the first sensor and the second sensor.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a limited-spectral component sensor; and
adjusting the attribute of the second image comprises normalizing values among the first image and the second image to substantially address a difference in sensitivity between the first sensor and the second sensor.

### Claim 6 (depends on 1)

The method of claim 1, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a luminance sensor; and
adjusting the attribute of the second image comprises normalizing luminance values among the first image and the second image to substantially address a difference in luminance sensitivity between the first sensor and the second sensor.

### Claim 7 (depends on 6)

The method of claim 6, further comprising generating a high dynamic range image by combining luminance values of the second image with the first image.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
generating a depth map for focusing the first image, the depth map including a mapping among relative depth values in a field of view based on a difference between pixels of the first image and pixels of the second image; and
generating an edge map by identifying edges in at least one of the first image or the second image.

### Claim 9

An image processing device, comprising:
a first sensor and a second sensor;
a memory coupled to the first sensor and the second sensor; and
a processing circuit coupled to the memory and configured to:
identify a characteristic for calibration associated with at least one of the first sensor or the second sensor;
capture a first image with the first sensor and capture a second image with the second sensor; and
adjust an attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the first image using the characteristic for calibration.

### Claim 10 (depends on 9)

The image processing device of claim 9, wherein the processing circuit is further configured to, before adjusting the attribute of the second image, adapt an operating characteristic of at least one of the first sensor or the second sensor to accommodate for at least one of noise, defective pixels, dark current, vignetting, demosaicing, or white balancing using the characteristic for calibration.

### Claim 11 (depends on 9)

The image processing device of claim 9, wherein the processing circuit is further configured to:
before adjusting the attribute of the second image, downscale the first image to a downscaled first image; and
adjust the attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the downscaled first image using the characteristic for calibration.

### Claim 12 (depends on 9)

The image processing device of claim 9, wherein the processing circuit is further configured to align the second image with the first image to substantially address a difference in alignment between the first sensor and the second sensor.

### Claim 13 (depends on 9)

The image processing device of claim 9, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a limited-spectral component sensor; and
the processing circuit is further configured to normalize values among the first image and the second image to substantially address a difference in sensitivity between the first sensor and the second sensor.

### Claim 14 (depends on 9)

The image processing device of claim 9, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a luminance sensor; and
the processing circuit is further configured to normalize luminance values among the first image and the second image to substantially address a difference in luminance sensitivity between the first sensor and the second sensor.

### Claim 15 (depends on 9)

The image processing device of claim 9, wherein the processing circuit is further configured to:
generate a depth map for focusing the first image, the depth map including a mapping among relative depth values in a field of view based on a difference between pixels of the first image and pixels of the second image; and
generate an edge map by identifying edges in at least one of the first image or the second image.

### Claim 16

An image processing method, comprising:
identifying a characteristic for calibration associated with at least one of a first sensor or a second sensor;
adapting an operating characteristic of at least one of the first sensor or the second sensor;
capturing a first image with the first sensor and capturing a second image with the second sensor; and
adjusting, with a processing circuit, an attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the first image using the characteristic for calibration.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
before adjusting the attribute of the second image, downscaling the first image to a downscaled first image, wherein:
adjusting the attribute of the second image comprises adjusting the attribute of the second image to substantially address a difference between the attribute of the second image and a corresponding attribute of the downscaled first image using the characteristic for calibration.

### Claim 18 (depends on 16)

The method of claim 16, wherein adjusting the attribute of the second image comprises aligning the second image with the first image to substantially address a difference in alignment between the first sensor and the second sensor.

### Claim 19 (depends on 16)

The method of claim 16, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a limited-spectral component sensor; and
adjusting the attribute of the second image comprises normalizing values among the first image and the second image to substantially address a difference in sensitivity between the first sensor and the second sensor.

### Claim 20 (depends on 16)

The method of claim 16, wherein:
the first sensor comprises a multi-spectral component sensor and the second sensor comprises a luminance sensor; and
adjusting the attribute of the second image comprises normalizing luminance values among the first image and the second image to substantially address a difference in luminance sensitivity between the first sensor and the second sensor.
