# US 2021/0174149 A1 · Feature fusion and dense connection-based method for infrared plane object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174149 A1
- Document ID: US20210174149A1
- Status: Published application
- Kind: A1
- Application number: 17/270,908
- Filed: 2018-11-20
- Published: 2021-06-10
- Earliest claimed: 2018-11-20
- Assignees: Xidian University
- Applicants: Xidian University
- Inventors: Huixin ZHOU; Jiajia ZHANG; Yuanbin SHI; Dong ZHAO; Lixin GUO; Hanlin QIN; Bingjian WANG; Rui LAI; Huan LI; Jiangluqi SONG; Bo YAO; Yue YU; Xiuping JIA; Jun ZHOU
- Web page: https://patentgrep.com/patent/US20210174149A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/253: {of extracted features}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06F 18/2413: based on distances to training or reference patterns
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A feature fusion and dense connection-based method for infrared plane object detection includes: constructing an infrared image dataset containing an object to be recognized, calibrating a position and class of the object to be recognized in the infrared image dataset, and obtaining an original known label image; dividing the infrared image dataset into a training set and a validation set; performing image enhancement preprocessing on images in the training set, performing feature extraction and feature fusion, and obtaining classification results and bounding boxes through a regression network; calculating a loss function according to the classification results and the bounding boxes in combination with the original known label image, and updating parameter values of a convolutional neural network; repeating the steps to iteratively update the parameters of the convolutional neural network; and processing images in the validation set through the parameters to obtain a final object detection result map.

## Claims

### Claim 1

A feature fusion and dense connection-based method for an infrared plane object detection, comprising the following steps:
step 1, constructing an infrared image dataset containing an object to be recognized, calibrating a position and a class of the object to be recognized in the infrared image dataset, and obtaining an original known label image;
step 2, dividing the infrared image dataset into a training set and a validation set;
step 3, performing an image enhancement preprocessing on images in the training set to obtain preprocessed images;
step 4, performing a feature extraction and a feature fusion on the preprocessed images, and obtaining classification results and bounding boxes through a regression network; calculating a loss function according to the classification results and the bounding boxes in combination with the original known label image, back-propagating a prediction error in a convolutional neural network by using a stochastic gradient descent method with momentum, and updating parameter values of the convolutional neural network;
step 5, repeating step 3 and step 4 to iteratively update the parameter values of the convolutional neural network until an error is small enough or a number of iterations reaches a set upper limit; and
step 6, processing images in the validation set through parameters of a trained convolutional neural network to obtain an accuracy and a required time for the infrared plane object detection, and a final object detection result map.

### Claim 2 (depends on 1)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 1, wherein
the step of performing the feature extraction and the feature fusion on the preprocessed images, and obtaining the classification results and the bounding boxes through the regression network in step 4 specifically comprises the following steps:
step 401, randomly extracting a fixed number of images from the training set, and dividing each image of the images from the training set into 10×10 cells;
step 402, inputting the each image divided in step 401 into a densely connected network for the feature extraction to obtain extracted feature maps;
step 403, performing the feature fusion on the extracted feature maps to obtain a fused feature map;
step 404, generating a fixed number of suggestion boxes for each cell of the 10×10 cells in the fused feature map; and
step 405, sending the fused feature map and the suggestion boxes to the regression network for a classification and a bounding box regression, and removing a redundancy by using a non-maximum suppression method to obtain the classification results and the bounding boxes.

### Claim 3 (depends on 2)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 2, wherein
a calculation method for the densely connected network in step 402 is expressed as the following formula:
dl=Hl([d0,d1, . . . ,dl-1]);
wherein, dl represents an output result of an lth convolutional layer in the densely connected network, and if the densely connected network contains a total of B convolutional layers, then a value of l is in a range of 0-B; Hl(*) represents a combined operation of regularization, a convolution and a linear rectification activation function; d0 represents an input image; and dl-1 represents an output result of an (l−1)th layer.

### Claim 4 (depends on 3)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 3, wherein
the step of performing the feature fusion on the extracted feature maps in step 403 comprises: directly fusing the extracted feature maps of different scales through a pooling method.

### Claim 5 (depends on 4)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 4, wherein
the step of performing the feature fusion on the extracted feature maps in step 403 specifically comprises the following steps:
step 4031, converting a first group of feature maps F1 into a new smaller feature map through a pooling operation, and then fusing the new smaller feature map with a second group of feature maps F2 to obtain a new feature map F2′;
step 4032, performing the pooling operation on the new feature map F2′, and then fusing with a third group of feature maps F3 to obtain a new feature map F3′; and
step 4033, replacing the second group of feature maps F2 and the third group of feature maps F3 with the new feature map F2′ and the new feature map F3′ to enter the regression network.

### Claim 6 (depends on 5)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 5, wherein
the step of sending the fused feature map and the suggestion boxes to the regression network for the classification and the bounding box regression, and removing the redundancy by using the non-maximum suppression method to obtain the classification results and the bounding boxes in step 405 specifically comprises the following steps:
step 4051, dividing the fused feature map into 10×10 cells, and inputting the fused feature map into a regression detection network;
step 4051, for each cell of the 10×10 cells of the fused feature map, outputting positions and classes of 7 possible objects by the regression detection network, wherein the 7 possible objects comprise A classes in total, probabilities corresponding to the A classes of the 7 possible objects are output, wherein A is related to a setting of the training set; and a position parameter comprises 3 pieces of data comprising center position coordinates, a width and a height of an object bounding box;
step 4052, according to the non-maximum suppression method, calculating an intersection-over-union of the bounding boxes of the same class by using the following formula:
S
=
M
⋂
N
M
⋃
N
;
wherein, S represents the intersection-over-union, M and N represent two bounding boxes of the same class of objects, M∩N represents an intersection of a bounding box M and a bounding box N, and M∩N represents a union of the bounding box M and the bounding box N; and
removing a bounding box with a smaller classification result value from the two bounding boxes with S greater than 0.75.

### Claim 7 (depends on 6)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 6, wherein
the step of calculating the loss function according to the classification results and the bounding boxes in combination with the original known label image, back-propagating the prediction error in the convolutional neural network by using the stochastic gradient descent method with the momentum, and updating the parameter values of the convolutional neural network in step 4 specifically comprises the following steps:
step 401, calculating the loss function according to the classification results, positions and classes of the 7 possible objects in the bounding boxes, and the position and the class of the object to be recognized in the training set by the following formula:
loss
=
∑
i
=
0
100

∑
j
=
0
7

∏
ij
obj

[
(
x
i
-
x
^
i
)
2
+
(
y
i
-
y
^
i
)
2
]
+
∑
i
=
0
100

∑
j
=
0
7

∏
ij
obj

[
(
w
i
-
w
^
i
)
2
+
(
h
i
-
h
^
i
)
2
]
+
∑
i
=
0
100

∑
j
=
0
7

∏
ij
obj

(
C
i
-
C
^
i
)
2
+
∑
i
=
0
100

∑
j
=
0
7

∏
ij
no


obj

(
C
i
-
C
^
i
)
2
wherein, 100 represents a number of the 10×10 cells, 7 represents a number of the suggestion boxes to be predicted for the each cell and the bounding boxes; i represents a numbering for the 10×10 cells; j represents a numbering for the suggestion boxes and the bounding boxes; loss represents an error value; obj indicates the object is present; noobj indicates the object is not present; x represents a predicted value of an abscissa of a center of each of the suggestion boxes and the bounding boxes, and y represents a predicted value of an ordinate of the center of each of the suggestion boxes and the bounding boxes; w represents a predicted value of a width of each of the suggestion boxes and the bounding boxes, and h represents a predicted value of a height of each of the suggestion boxes and the bounding boxes; C represents a predicted value indicating whether the suggestion boxes and the bounding boxes contain the object, and C comprises A values, wherein the A values correspond to the probabilities of the A classes of the 7 possible objects, respectively; {circumflex over (x)}, ŷ, ŵ, ĥ, Ĉ represent corresponding label values; Πijobj indicates that the object falls into a jth suggestion box and a jth bounding box of a cell i, and Πijnoobj indicates that the object does not fall into the jth suggestion box and the jth bounding box of the cell i; and
step 402, according to a calculation result of the loss function, updating weights by using the stochastic gradient descent method with the momentum.

### Claim 8 (depends on 1)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 1, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 9 (depends on 2)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 2, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 10 (depends on 3)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 3, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 11 (depends on 4)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 4, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 12 (depends on 5)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 5, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 13 (depends on 6)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 6, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.

### Claim 14 (depends on 7)

The feature fusion and dense connection-based method for the infrared plane object detection according to claim 7, wherein
the image enhancement preprocessing in step 3 comprises: augmenting the training set through a random rotation, mirroring, flipping, scaling, a translation, a scale transformation, a contrast transformation, a noise disturbance and a color change.
