# US 10,504,245 B2 · Image processing device and method for color balancing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,245 B2
- Document ID: US10504245B2
- Status: Granted
- Kind: B2
- Application number: 15/651,673
- Filed: 2017-07-17
- Published: 2017-11-02
- Granted: 2019-12-10
- Earliest claimed: 2015-03-18
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Karim Helwani; Lukasz Kondrad; Nicola Piotto
- Web page: https://patentgrep.com/patent/US10504245B2

## CPC

- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 7/90: Determination of colour characteristics
- H04N 1/486: {with separate detectors, each detector being used for one specific colour component}
- H04N 1/6077: {Colour balance, e.g. colour cast correction}
- H04N 1/6086: {by scene illuminant, i.e. conditions at the time of picture capture, e.g. flash, optical filter used, evening, cloud, daylight, artificial lighting, white point measurement, colour temperature}
- H04N 13/243: using three or more two-dimensional [2D] image sensors
- H04N 13/246: Calibration of cameras
- H04N 13/257: Colour aspects
- H04N 17/002: {for television cameras}
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/10024: Color image
- G06T 2207/10152: Varying illumination
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An image processing device performing color balancing of a first image and at least a second image is provided. The image processing device comprises a color balancing determination unit and a color balancing calculation unit. The color balancing determination unit determines a global gain vector (t) comprising at least two gain factors (ân, ân+1) of the first and second images by minimizing a cost function based upon reference pixel values of the first and second images. The first and second image reference pixels depict a shared color scene of the two images. The color balancing calculation unit performs color balancing of the first image based upon the gain factor (ân) of the first image and to perform a color balancing of the second image based upon the gain factor (ân+1) of the second image.

## Claims

### Claim 1

An image processing device for performing a color balancing of at least a first image recorded by a first camera and a second image recorded by a second camera, the image processing device comprising one or more processors adapted to:
determine a global gain vector (t) comprising at least one gain factor (ân) of the first image and at least one gain factor (ân+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images;
perform a color balancing of the first image based upon the at least one gain factor (ân) of the first image; and
perform a color balancing of the second image based upon the at least one gain factor (ân+1) of the second image,
wherein minimizing the pre-determined cost function comprises minimizing
J′=[yj(1) yj(2)][a(2) −a(1)]T,
wherein J′ is the predetermined cost function, T is a transpose, j is a pixel number, yj(1) is a value of pixel number j measured by the first camera, y1(2) is a value of pixel number j measured by the second camera, a(1) is a gain factor by which a reference color value of the pixel j has to be multiplied to equal yj(1), and a(2) is a gain factor by which the reference color value of the pixel j has to be multiplied to equal yj(2).

### Claim 2 (depends on 1)

The image processing device according to claim 1,
wherein the first image and the second image each comprise at least three color channels, and wherein the one or more processors are further adapted to:
minimize separate cost functions for each color channel of the images;
perform the color balancing of the first image separately for each color channel; and
perform the color balancing of the second image separately for each color channel.

### Claim 3 (depends on 1)

The image processing device according to claim 1,
wherein the shared color scene of the first image and the second image is a scene that captures at least one same object or area, and
wherein the at least one same object or area is within a shared field of view of the first camera and the second camera.

### Claim 4 (depends on 1)

The image processing device according to claim 1,
wherein the shared color scene of the first image and the second image is a scene that captures at least one same color, and
wherein the at least one same color is not within a shared field of view of the first camera and the second camera.

### Claim 5 (depends on 1)

The image processing device according to claim 1,
wherein the image processing device is adapted to perform a color balancing of at least the first image recorded by the first camera, the second image recorded by the second camera, and a third image recorded by a third camera,
wherein the one or more processors, after performing the color balancing of the first image and the second image, resulting in a color balanced first image and a color balanced second image, are further adapted to:
determine a further global gain vector comprising at least one further gain factor of the second image and at least one gain factor of the third image by minimizing a pre-determined cost function based upon reference pixel values of the second color balanced image and reference pixel values of the third image, wherein the reference pixels of the color balanced second image and the third image depict a shared color scene of the color balanced second image and the third image;
perform a color balancing of the second color balanced image based upon the at least one further gain factor of the second image; and
perform a color balancing of the third image based upon the at least one gain factor of the third image.

### Claim 6 (depends on 1)

The image processing device according to claim 1,
wherein a global gain vector t is assumed to achieve
J=Y·t, where
Y=[y(1) y(2)],
y(n)=[y1(n) . . . yi(n) . . . y1(n)]T, and
t=[a(2) −a(1)]T,
wherein t is the global gain vector, Y is a vector of values y measured for a single pixel by first camera and the second camera, n is a pixel number, i is an intermediary pixel number, and 1 is a last pixel number, and minimizing the pre-determined cost function comprises
determining a non-trivial nullspace of a matrix
Ryy=YT·Y by preferably performing an eigenvalue decompositon of Ryy
Ryy=·Ryy·ÂT,
wherein Ryy is a diagonal matrix with the eigenvalues at its main diagonal, and  is a unitary matrix, and
using the eigenvectors with the smallest eigenvalues {circumflex over (t)} as the first color balancing information â(1) and the second color balancing information â(2), wherein
{circumflex over (t)}=[â(2) −â(1)]T.

### Claim 7 (depends on 6)

The image processing device according to claim 6,
wherein performing of the color balancing comprises:
dividing pixel values yj(1) of the first camera by the gain factor â(1), and
dividing pixel values yj(1) of the second camera by the gain factor â(2).

### Claim 8

An imaging system comprising:
a first camera;
a second camera; and
an image processing device comprising one or more processors adapted to:
determine a global gain vector (t) comprising at least one gain factor (ân) of the first image and at least one gain factor (ân+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images;
perform a color balancing of the first image based upon the at least one gain factor (ân) of the first image; and
perform a color balancing of the second image based upon the at least one gain factor (ân+1) of the second image,
wherein minimizing the pre-determined cost function comprises minimizing
J′=[yj(1) yj(2)][a(2) −a(1)]T,
wherein J′ is the predetermined cost function T is a transpose j is a pixel number yj(1) is a value of pixel number j measured by the first camera, y1(2) is a value of pixel number j measured by the second camera, a(1) is a gain factor by which a reference color value of the pixel j has to be multiplied to equal yj(1) and a(2) is a gain factor by which the reference color value of the pixel j has to be multiplied to equal yj(2).

### Claim 9 (depends on 8)

The imaging system according to claim 8,
wherein the first camera and the second camera have an at least partially shared field of view.

### Claim 10 (depends on 8)

The imaging system according to claim 8, further comprising a third camera.

### Claim 11 (depends on 10)

The imaging system according to claim 10,
wherein each of the cameras has at least a partially shared field of view with at least one other of the cameras.

### Claim 12

An image processing method for performing a color balancing of at least a first image recorded by a first camera and a second image recorded by a second camera,
the method comprising:
determining a global gain vector (t) comprising at least one gain factor (ân) of the first image and at least one gain factor (ân+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images;
performing a color balancing of the first image based upon the at least one gain factor (ân) of the first image; and
performing a color balancing of the second image based upon the at least one gain factor (ân+1) of the second image,
wherein minimizing the pre-determined cost function comprises minimizing
J′=[yj(1) yj(2)][a(2) −a(1)]T,
wherein J′ is the predetermined cost function, T is a transpose, j is a pixel number, yj(1) is a value of pixel number j measured by the first camera y1(2) is a value of pixel number j measured by the second camera, a(1) is a gain factor by which a reference color value of the pixel j has to be multiplied to equal yj(1) and a(2) is a gain factor by which the reference color value of the pixel j has to be multiplied to equal yj(2).

### Claim 13

A non-transitory computer readable medium comprising a program code for performing a color balancing of at least a first image recorded by a first camera and a second image recorded by a second camera when the program code runs on a computer, the computer is configured to perform steps comprising:
determining a global gain vector (t) comprising at least one gain factor (ân) of the first image and at least one gain factor (ân+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images;
performing a color balancing of the first image based upon the at least one gain factor (ân) of the first image; and
performing a color balancing of the second image based upon the at least one gain factor (ân+1) of the second image,
wherein minimizing the pre-determined cost function comprises minimizing
J′=[yj(1) yj(2)][a(2) −a(1)]T,
wherein J′ is the predetermined cost function, T is a transpose, j is a pixel number, yj(1) is a value of pixel number j measured by the first camera, y1(2) is a value of pixel number j measured by the second camera, a(1) is a gain factor by which a reference color value of the pixel j has to be multiplied to equal yj(1), and a(2) is a gain factor by which the reference color value of the pixel j has to be multiplied to equal yj(2).