# US 10,504,239 B2 · Methods and systems for camera characterization in terms of response function, color, and vignetting under non-uniform illumination

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,504,239 B2
- Document ID: US10504239B2
- Status: Granted
- Kind: B2
- Application number: 15/566,536
- Filed: 2016-04-13
- Published: 2018-03-29
- Granted: 2019-12-10
- Earliest claimed: 2015-04-13
- Assignees: Universidade De Coimbra
- Applicants: Universidade De Coimbra
- Inventors: Pedro Miguel Cerveira Rodrigues; João Pedro de Almeida Barreto
- Web page: https://patentgrep.com/patent/US10504239B2

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/90: Determination of colour characteristics
- G06T 15/506: {Illumination models}
- H04N 1/60: Colour correction or control {(H04N1/54 takes precedence)}
- H04N 1/6019: {using look-up tables (H04N1/6025 takes precedence)}
- H04N 17/02: for colour television signals
- H04N 25/61: the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
- G06T 2207/10024: Color image
- G06T 2207/10068: Endoscopic image
- G06T 2207/10152: Varying illumination
- G06T 2207/30244: Camera pose
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Methods and systems for describing a camera radiometrically, in terms of camera response function and vignetting, and in terms of color, suitable for non-uniform illumination set-ups. It estimates the camera response function and the camera color mapping from a single image of a generic scene with two albedos. With a second same-pose image with a different intensity of the near-light the vignetting is also estimated. The camera response function calibration involves the segmentation of the two albedos, the definition of the system of equations based on the assumption that can be made about the image formation process, and the actual estimation. For the modelling the vignetting there are three steps: computing the albedo-normalized irradiance, finding points of equal vignetting, when needed, and estimation.

## Claims

### Claim 1

A method for estimating a color model of an imaging device, comprised of the response function and the color mapping, under generic illumination, the method comprising:
i. acquiring at least one calibration image of a scene with at least two regions of constant albedo from an arbitrary pose;
ii. segmenting the calibration image into regions of constant albedo and grouping the pixels in each region into a set , the set of points with albedo ρn;
iii. deriving equations for the estimation using
f−1(d(x))=u(x)ρ′(x)
where x are the pixel coordinates that correspond to a scene point, d(x) is the acquired image and is defined as a vector of Nc elements dc(x) that correspond to values on pixel x and channel c, f−1(d) is the inverse camera response function, a function vector of elements f c−1(dc), u(x) is the albedo-normalized irradiance, a scalar function, and ρ′ is the albedo vector of size Nc in the color space of the camera and is defined by elements ρ′c for each channel c such that
ρ=Mρ′
with ρ being the albedo vector of size Ns in the output color space and is defined by elements ρs for each channel s, and M being a color mapping matrix of size Ns by Nc and composed of rows msT; and
iv. determining the color model of the imaging device by solving the equations derived in step iii using a suitable solver or optimization scheme.

### Claim 2 (depends on 1)

The method of claim 1, wherein the scene used in step i is a known planar checkerboard pattern.

### Claim 3 (depends on 1)

The method of claim 1, wherein Nc=1 for single-channel imaging devices, Nc=3 or trichromatic imaging devices, Nc>3 for multispectral and hyperspectral imaging devices, and Ns=Nc for an output image with the same number of channels as the input image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the equations derived in step iii are defined directly from the image formation model as
fc−1(dc(x))=u(x)ρ′c(x)
or, equivalently, in the logarithmic space using
log fc−1(dc(x))=log u(x)+log ρ′c(x)
where log stands for a logarithm of any base and where d(x) is known and f−1, u(x), and ρ′(x) are optimized for in step iv.

### Claim 5 (depends on 4)

The method of claim 4, wherein the scene contains at least Nc albedos of known values on the output color space, which enables the estimation of M after the estimation of f−1,u(x), and ρ′(x) using equations of the form
ρs=msTρ′
or together with the estimation of f−1 and u(x) using equations of the form
msTf−1(d(x))=u(x)ρs(x).

### Claim 6 (depends on 1)

The method of claim 1, wherein the equations derived in step iii are defined on the isocurves of u(x), in which case the d(x) is known and f−1 is optimized for in step iv.

### Claim 7 (depends on 6)

The method of claim 6, wherein to estimate the isocurves of u(x), a parametric model h(x) is fitted to the regions of one albedo or more than one independently or using, for the purpose of incorporating more than one albedo on the same model and estimate only one set of isocurves, the following equations:
h
⁡
(
x
j
)
=
d
c
⁡
(
x
j
)
h
⁡
(
x
k
)
=
g
c
,
c
′
,
n
,
n
′
⁡
(
d
c
′
⁡
(
x
k
)
)
,
j
∈
𝒜
ρ
n
,
k
∈
𝒜
ρ
n
′
for a pair of albedos ρn and ρn′, where gc,c′,n,n′ is a function that maps the values of albedo ρc′,n′ to the model of albedo ρ′c,n,h(x), and where only d(x) is known.

### Claim 8 (depends on 6)

The method of claim 6, wherein the isocurves are used to establish equations of the form
ρ′c′,n′fc−1(dc(xj))=ρ′c,nfc′−1(dc′(xk)), j∈i∩k∈i∩
for a pair of albedos ρn and ρn′, where i is the set of pixels crossed by isocurve i, and where d(x) is known and f−1 and ρ′(x) are optimized for in step iv.

### Claim 9 (depends on 8)

The method of claim 8, wherein the scene contains at least N calbedos of known values on the output color space, which enables the estimation of M after the estimation of f−1 and ρ′(x) using equations of the form
ρs=msTρ′
or together with the estimation of f−1 using equations of the form
ρs′,n′msTf−1(d(xj))=ρs,nms′Tf−1(d(xk)),j∈i∩,k∈i∩.

### Claim 10 (depends on 1)

The method of claim 1, wherein the equation derivation in step iii uses a particular case matrix for the color mapping matrix comprising a diagonal matrix or a rotation matrix.

### Claim 11 (depends on 1)

The method of claim 1, wherein the solution of step iv is found using contributions from multiple calibration images, in which case steps i to iii are executed for each one of these images.

### Claim 12 (depends on 1)

The method of claim 1, wherein the determination of the color model in step iv is carried together with one or more of monotonicity constraints on the camera response function and regularization of search parameters.

### Claim 13 (depends on 1)

The method of claim 1, wherein for the purpose of having a camera response function defined for the whole range of possible pixel values d, a parametric function, an empiric model of response, or an interpolation/extrapolation method is used together with or as post processing to solving the system of equations in step iv.

### Claim 14 (depends on 1)

The method of claim 1, wherein the optimization in step iv is executed with nonlinear optimization algorithms, comprising interior point, gradient descent, and metaheuristics, for the cases of systems of equations that are nonlinear, and it is executed with convex optimization algorithms, comprising quadratic programming, for the cases of system of equations that are linear.

### Claim 15 (depends on 1)

The method of claim 1, wherein the color model of the imaging device is used to normalize the response function, by computing a new image where the pixel values d′(x) are given by
d′=f−1(d)
for image points x or, in the case of the color mapping matrix being also known, are given by
d′=Mf1(d).

### Claim 16 (depends on 15)

The method of claim 15, wherein the new image is computed for improved visualization and/or for further processing by other methods or algorithms, comprising shape-from-shading, photometric vision algorithms, computer-aided surgery methods, and transformations of pixel values for enhancing image properties or features, such as contrast or particular colors.

### Claim 17 (depends on 1)

The method of claim 1, wherein the output color space is a standard color space, comprising CIEXYZ and sRGB, or other predetermined color space, comprising the color space of another imaging device.

### Claim 18 (depends on 1)

The method of claim 1, wherein the algorithms presented may be used as a starting point of, integrated in, or complement other models of imaging devices, comprising multi-dimensional look-up tables.

### Claim 19

A method for estimating a vignetting model of an imaging device comprising:
i. acquiring at least one set of at least two calibration images with the same relative pose and a different intensity of a near-light source;
ii. segmenting the calibration images into regions of constant albedo;
iii. computing the albedo-normalized irradiance for each pair of same-pose images, u1(x) and u2(x);
iv. deriving the equations of the vignetting model using
m(x)=K1u1(x)+K2u2(x)
where K1 and K2 are the two model unknowns; and
v. finding a solution for the vignetting model with the equations created in step iv.

### Claim 20 (depends on 19)

The method of claim 19, wherein, for the purpose of having additional equations, sets of points with equal vignetting are found in step iv by searching for points where one or more of both u1(x)/u2(x) and u1(x) are constant and both u1(x)/u2(x) and u2(x) are constant, which may be performed either by exhaustive search or by intersecting the isocurves of one or more of u1(x)/u2(x) and u1(x) and u1(x)/u2(x) and u2(x).

### Claim 21 (depends on 19)

The method of claim 19, wherein the vignetting estimation equations in step iv, when two or more pairs of same-pose images are available can be defined as
m(x)=k1u1(x)+K2u2(x)
m(x)=K′1u′1(x)+k′2u′2(x)
for every two pairs, which can be rewritten as
k1u1(x)+K2u2(x)=K′1u′1(x)+k′2u2(x)
and, in this way, creating a system of equations from which the vignetting model unknowns (K1, K2, K′1, and k′2) can be estimated.

### Claim 22 (depends on 19)

The method of claim 19, wherein the vignetting model of the imaging device is used to normalize any acquired images and videos, for improved visualization and/or for further processing by other methods.