# US 2022/0044375 A1 · Saliency Map Enhancement-Based Infrared and Visible Light Fusion Method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044375 A1
- Document ID: US20220044375A1
- Status: Published application
- Kind: A1
- Application number: 17/283,181
- Filed: 2020-03-05
- Published: 2022-02-10
- Earliest claimed: 2019-12-17
- Assignees: Dalian University of Technology
- Applicants: Dalian University of Technology
- Inventors: Risheng LIU; Xin FAN; Jinyuan LIU; Wei ZHONG; Zhongxuan LUO
- Web page: https://patentgrep.com/patent/US20220044375A1

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 5/20: using local operators
- G06T 5/80: Geometric correction
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20024: Filtering details
- G06T 2207/20164: Salient point detection; Corner detection
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present invention proposes a saliency map enhancement-based infrared and visible light fusion method, which is an infrared and visible light fusion algorithm using filtering decomposition and significant enhancement. Binocular cameras and NVIDIA TX2 are used to construct a high-performance computing platform and to construct a high-performance solving algorithm to obtain a high-quality infrared and visible light fusion image. The system is easy to construct, and the input data can be acquired by using stereo binocular infrared and visible light cameras respectively; the program is simple and easy to implement; the input image is decomposed into a background layer and a detail layer by means of filtering decomposition according to different imaging principles of infrared and visible light cameras, a saliency map enhancement-based fusion method is designed for the background layer, and a pixel contrast-based fusion algorithm is designed for the detail layer.

## Claims

### Claim 1

A saliency map enhancement-based infrared and visible light fusion method, wherein the method comprises the following steps:
1) obtaining registered infrared and visible light images, and respectively calibrating each lens and jointly calibrating the respective systems of the visible light binocular camera and the infrared binocular camera;
1-1) respectively calibrating the infrared camera and the visible light camera by the Zhangzhengyou calibration method to obtain internal parameters including focal length and principal point position and external parameters including rotation and translation of each camera;
1-2) calculating the positional relationship of the same plane in the visible light image and the infrared image by using the pose relationship RT of the visible light camera and the infrared camera obtained by joint calibration of the cameras and the detected checker corners, and registering the visible light image to the infrared image by using homography transformation;
2) converting the color space of the visible light image from an RGB image to an HSV image, extracting the value information of the color image as the input of image fusion, and retaining the original hue and saturation;
3) carrying out mutual guided filtering decomposition on the input infrared image and the visible light image with the color space converted, and decomposing the images into a background layer and a detail layer, wherein the background layer includes the structural information of the images, and the detail layer includes the gradient and texture information of the images;
B=M(I,V), D=(I,V)−B
wherein B represents the background layer, D represents the detail layer, M represents the mutual guided filtering, and I represents the infrared image;
4) fusing the background layer B by the saliency map-based method, subtracting each pixel from all the pixels, and taking and adding absolute values, wherein the formula is as follows:
S(p)=|I(p)−I1|+|I(p)−I2|+|I(p)−I3|+ ⋅ ⋅ ⋅ +|I(p)−IN|
that is
S
⁡
(
p
)
=
∑
i
=
0
2
⁢
5
⁢
5
⁢
M
⁡
(
i
)
⁢

I
⁡
(
p
)
-
i

wherein S(p) represents the salience value of the pixel, N represents the number of pixels in the image, M represents the histogram statistical formula, and I(p) represents the value of the pixel position;
according to the obtained saliency value, obtaining the saliency map weight based on background layer fusion:
W
=
S
⁡
(
p
)
∑
S
⁡
(
p
)
j
wherein W represents the weight, and Sj represents the corresponding pixel value; then performing linearly weighted fusion of the decomposed infrared image and visible light image based on the saliency map weight, wherein the calculation formula is as follows:
B=0.5*(0.5+I*(W1−W2)*0.5)+0.5*(0.5+V*(W2−W1)*0.5)
wherein I and V represent the input infrared image and visible light image respectively, and W1 and W2 represent the salience weights obtained for the infrared image and visible light image respectively;
5) implementing the contrast-based pixel fusion strategy for the detail layer obtained after object difference, designing a sliding window to perform global sliding on the detail images of infrared and visible light respectively, comparing the pixel values of the corresponding detail images, and taking 1 if the values of eight neighborhoods of the current pixel of the infrared image are greater than those of eight neighborhoods of the corresponding pixel of the visible light; otherwise, taking 0; generating the corresponding binary weight map X according to the scanned sliding window; and then fusing the detail layer;
D=D(I)*X+D(V)*(1−X)
6) linearly weighting the background layer and the detail layer to obtain:
F=B+D
wherein F represents the fusion result, and B and D represent the background layer fusion result and detail layer fusion result;
7) converting the color space: converting the fusion image back to the RGB image, and adding the hue and saturation previously retained;
restoring to the RGB color space from the HSV color space by updating the V information saved into the fusion image in combination with the previously retained H and S;
8) enhancing the color: enhancing the color of the fusion image to generate a fusion image with higher resolution and contrast; and performing pixel-level image enhancement for the contrast of each pixel;
performing color correction and enhancement on the restored image to generate a three-channel image that is consistent with observation and detection; and performing color enhancement on the R channel, G channel and B channel respectively to obtain the final fusion image.

### Claim 2 (depends on 1)

The saliency map enhancement-based infrared and visible light fusion method according to claim 1, wherein the color space conversion of the visible light image in step 2) comprises:
2-1) converting the RGB color space to the HSV color space, wherein V is value, H is hue and S is saturation; and extracting the value information of the visible light image to be fused with the infrared image, and retaining the hue and saturation, wherein the specific conversion is shown as follows:
R′=R/255 G′=G/255 B′=B/255
Cmax=max(R′,G′,B′)
Cmin=min(R′,G′,B′)
Δ=Cmax−Cmin
V=Cmax
2-2) extracting the V channel as the input of visible light, retaining H and S to the corresponding matrix, and retaining the color information for the subsequent color restoration after fusion.

### Claim 3 (depends on 1)

The saliency map enhancement-based infrared and visible light fusion method according to claim 1, wherein the specific formulas for color space conversion in step 7) are shown as follows:
C
=
V
×
S
X
=
C
×
(
1
-

(
H
⁢
/
⁢
60
⁢
°
)
⁢
⁢
mod
⁢
⁢
2
-
1

)
m
=
V
-
C
⁢
(
R
′
,
G
′
,
B
′
)
=
{
(
C
,
X
,
0
)
⁢
,
0
⁢
°
≤
H
<
60
⁢
°
(
X
,
C
,
0
)
⁢
,
60
⁢
°
≤
H
<
120
⁢
°
(
0
,
C
,
X
)
,
120
⁢
°
≤
H
<
180
⁢
°
(
0
,
X
,
C
)
,
180
⁢
°
≤
H
<
240
⁢
°
(
X
,
0
,
C
)
⁢
240
⁢
°
≤
H
<
300
⁢
°
(
C
,
0
,
X
)
,
300
⁢
°
≤
H
<
360
⁢
°
⁢
⁢
R
′
,
G
′
,
B
′
=
(
(
R
′
+
m
)
×
2
⁢
55
⁢
,
⁢
(
G
′
+
m
)
×
2
⁢
55
⁢
,
(
B
′
+
m
)
×
2
⁢
5
⁢
5
)
wherein C is the product of the value and the saturation; and in is the difference of the value and C.

### Claim 4 (depends on 1)

The saliency map enhancement-based infrared and visible light fusion method according to claim 1, wherein the specific formulas for color enhancement in step 8) are shown as follows:
Rout=(Rin)1/gamma
Rdisplay=(Rin(1/gamma))gamma
Gout=(Gin)1/gamma
G=(Gin(1/gamma))gamma
Bout=(Bin)1/gamma
Bdisplay=(Bin(1/gamma))gamma
wherein gamma is the correction parameter, Rin, Gin and Bin are the values of the three input channels R, G, and B respectively, Rout, Gout and Bout are the intermediate parameters, and Rdisplay, Gdisplay and Bdisplay are the values of the three channels after enhancement.