# US 12,183,060 B2 · Method and apparatus for extreme-light image enhancement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,060 B2
- Document ID: US12183060B2
- Status: Granted
- Kind: B2
- Application number: 17/894,974
- Filed: 2022-08-24
- Published: 2023-05-11
- Granted: 2024-12-31
- Earliest claimed: 2021-11-05
- Assignees: Vinai Artificial Intelligence Application and Research Joint Stock Company
- Applicants: Vinai Artificial Intelligence Application and Research Joint Stock Company
- Inventors: Thi Hue Nguyen; Thi Ngoc Diep Tran; Cong Thanh Tran; Duc Minh Khoi Nguyen; Ho Man Rang Nguyen; Hai Hung Bui
- Web page: https://patentgrep.com/patent/US12183060B2

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06V 10/141: Control of illumination
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/56: relating to colour
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In a method and apparatus for extreme-light image enhancement, an input image is passed to a lighting classification module in order to be classified into two categories: an under-exposed lighting condition and an over-exposed light condition. Then the classified image is subsequently passed to an enhancement module which comprises a first image enhancement unit and a second image enhancement unit using a mapping function based on a gamma mapping function for mapping the classified image to an output image, respectively. A parameter of the gamma mapping function is estimated by a parameter estimation network, based on the classified image. The parameter estimation network may be a lightweight CNN network which is trained by an unsupervised approach.

## Claims

### Claim 1

A method for extreme-light image enhancement, comprising:
classifying, by a lighting classification module, an input image into a classified image, which corresponding to one of at least a first lighting condition and a second light condition;
receiving, by an image enhancement module, the classified image, wherein the image enhancement module comprises at least a first image enhancement unit and a second image enhancement unit, wherein each of the first image enhancement unit and the second image enhancement unit, using a mapping function which is based on a gamma mapping function and a parameter estimation network which estimates a parameter of the corresponding mapping function, maps the classified image to an output image; and
updating, by the corresponding parameter estimation network, an estimated parameter for the mapping function of an image enhancement unit among the first image enhancement unit and the second image enhancement unit,
wherein the image enhancement module is configured to perform:
mapping, by the first image enhancement unit of the image enhancement module, the classified image to the output image, if the classified image is corresponding to the first lighting condition; and
mapping, by the second image enhancement unit of the image enhancement module, the classified image to the output image, if the classified image is corresponding to the second lighting condition,
wherein the mapping function is defined by the following equation: Y=1−(1−I)y, wherein Y denotes the output image, I denotes the input image, and y denotes the parameter of the gamma mapping function, and
wherein said classifying comprises:
extracting a histogram information from the input image; and
detecting peaks of the histogram information by fitting a Gaussian Mixture Model to the histogram information.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the first lighting condition is an under-exposed lighting condition, and the second lighting condition is an over-exposed lighting condition.

### Claim 3 (depends on 1)

The method according to claim 1, wherein said classifying further comprises:
determining whether the input image is corresponding to the under-exposed lighting condition or the over-exposed light condition, based on the peaks of the histogram information.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the parameter estimation network uses a U-Net convolutional neural network architecture, and wherein each basic block in said U-Net network architecture is an inverted residual and linear bottleneck block, and is designed on a basic of a MobileNet network.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the parameter estimation network is trained by using a loss function defined by the following equation:
L=wcolLcol+wexpLexp+wconLcon+wtvLtv
wherein:
wcol, wexp, wcon, wtv are coefficients to balance between a color balance, an exposure correction, a contrast preservation, and a smoothness of the mapping function, respectively, and
Lcol is a color constancy loss function, Lexp is an exposure loss function, Lcon is a contrast preserving loss function, and Ltv is a total variation loss function.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the exposure loss function is defined by the following equation:
L
e
⁢
x
⁢
p
=
1
M
⁢
∑
i
=
1
M
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
Y
i
-
E
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
2
,
wherein the color constancy loss function is defined by the following equation:
Lcol=Σ∀(p,q)∈C(Jp−Jq)2 for C∈{(R,B),(B,G),(G,R)},
wherein the contrast preserving loss function is defined by the following equation:
L
c
⁢
o
⁢
n
=
1
W
⁢
H
⁢
∑
i
∑
j
C
x
(
I
i
,
j
)
⁢
(
C
x
(
I
i
,
j
)
-
C
x
(
Y
i
,
j
)
)
2
+
C
y
(
I
i
,
j
)
⁢
(
C
y
(
I
i
,
j
)
-
C
y
(
Y
i
,
j
)
)
2
,
wherein the total variation loss function is defined by the following equation:
L
t
⁢
v
=
1
N
⁢
∑
c
∈
R
,
G
,
B
(
∇
x
γ
c
+
∇
y
γ
c
)
,
and
wherein:
C
x
(
I
i
,
j
)
=
❘
"\[LeftBracketingBar]"
I
i
,
j
-
I
i
+
1
,
j
❘
"\[RightBracketingBar]"
1
-
min
⁢
(
I
i
,
j
,
I
i
+
1
,
j
)
,
C
y
(
I
i
,
j
)
=
❘
"\[LeftBracketingBar]"
I
i
,
j
-
I
i
,
j
+
1
❘
"\[RightBracketingBar]"
1
-
min
⁢
(
I
i
,
j
,
I
i
,
j
+
1
)
,
and
R, G, B denote the red, green, and blue channels of the input image.

### Claim 7 (depends on 1)

A non-transitory computer readable storage medium storing a computer readable program, wherein the computer readable program is configured to implement the method according to claim 1.

### Claim 8

An apparatus for extreme-light image enhancement, comprising:
one or more processors;
a non-transitory computer-readable medium having instructions stored there on, which, when executed by the one or more processors, cause the apparatus to execute one or more components, wherein the one or more components comprise:
a lighting classification module for classifying an input image into a classified image, which corresponding to one of at least a first lighting condition and a second light condition; and
an image enhancement module for receiving the classified image, wherein the image enhancement module comprises at least a first image enhancement unit and a second image enhancement unit,
wherein each of the first image enhancement unit and the second image enhancement unit, using a mapping function which is based on a gamma mapping function and a parameter estimation network which estimates a parameter of the corresponding mapping function, maps the classified image to an output image,
wherein a corresponding parameter estimation network updates an estimated parameter for a mapping function of an image enhancement unit among the first image enhancement unit and the second image enhancement unit,
wherein the image enhancement module is configured so that:
the first image enhancement unit of the image enhancement module maps the classified image to the output image, if the classified image is corresponding to the first lighting condition;
and
the second image enhancement unit of the image enhancement module maps the classified image to the output image, if the classified image is corresponding to the second condition,
wherein the mapping function is defined by the following equation: Y=1−(1−I)y, wherein Y denotes the output image, I denotes the input image, and y denotes the parameter of the gamma mapping function, and
wherein the lighting classification module is configured to:
extracting a histogram information from the input image; and
detect peaks of the histogram information by fitting a Gaussian Mixture Model to the histogram information.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein the first lighting condition is an under-exposed lighting condition, and the second lighting condition is an over-exposed lighting condition.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein the lighting classification module is further configured to:
determine whether the input image is corresponding to the under-exposed lighting condition or the over-exposed light condition, based on the peaks of the histogram information.

### Claim 11 (depends on 8)

The apparatus according to claim 8, wherein the parameter estimation network uses a U-Net convolutional neural network architecture, and wherein each basic block in said U-Net network architecture is an inverted residual and linear bottleneck block, and is designed on a basic of a MobileNet network.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the parameter estimation network is trained by using a loss function defined by the following equation:
L=wcolLcol+wexpLexp+wconLon+wtvLtv
wherein:
wexp, wcon, wtv are coefficients to balance between a color balance, an exposure correction, a contrast preservation, and a smoothness of the mapping function, respectively, and
Lcol is a color constancy loss function, Lexp is an exposure loss function, Lcon is a contrast preserving loss function, and Ltv is a total variation loss function.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the exposure loss function is defined by the following equation:
L
e
⁢
x
⁢
p
=
1
M
⁢
∑
i
=
1
M
❘
"\[LeftBracketingBar]"
❘
"\[LeftBracketingBar]"
Y
i
-
E
❘
"\[RightBracketingBar]"
❘
"\[RightBracketingBar]"
2
2
,
wherein the color constancy loss function is defined by the following equation:
Lcol=Σ∀(p,q)∈C(Jp−Jq)2 for C∈{(R,B),(B,G),(G,R)},
wherein the contrast preserving loss function is defined by the following equation:
L
c
⁢
o
⁢
n
=
1
W
⁢
H
⁢
∑
i
∑
j
C
x
(
I
i
,
j
)
⁢
(
C
x
(
I
i
,
j
)
-
C
x
(
Y
i
,
j
)
)
2
+
C
y
(
I
i
,
j
)
⁢
(
C
y
(
I
i
,
j
)
-
C
y
(
Y
i
,
j
)
)
2
,
wherein the total variation loss function is defined by the following equation:
L
t
⁢
v
=
1
N
⁢
∑
c
∈
R
,
G
,
B
(
∇
x
γ
c
+
∇
y
γ
c
)
,
and
wherein:
C
x
(
I
i
,
j
)
=
❘
"\[LeftBracketingBar]"
I
i
,
j
-
I
i
+
1
,
j
❘
"\[RightBracketingBar]"
1
-
min
⁢
(
I
i
,
j
,
I
i
+
1
,
j
)
,
C
y
(
I
i
,
j
)
=
❘
"\[LeftBracketingBar]"
I
i
,
j
-
I
i
,
j
+
1
❘
"\[RightBracketingBar]"
1
-
min
⁢
(
I
i
,
j
,
I
i
,
j
+
1
)
,
and
R, G, B denote the red, green, and blue channels of the input image.