# US 12,002,254 B2 · Method and apparatus of training object detection network and object detection method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,254 B2
- Document ID: US12002254B2
- Status: Granted
- Kind: B2
- Application number: 17/613,442
- Filed: 2021-02-26
- Published: 2022-09-01
- Granted: 2024-06-04
- Earliest claimed: 2021-02-26
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: BOE Technology Group Co., Ltd.
- Inventors: Jingru Wang; Fengshuo Hu
- Web page: https://patentgrep.com/patent/US12002254B2

## CPC

- G06V 10/56: relating to colour
- G06T 7/73: using feature-based methods
- G06T 7/90: Determination of colour characteristics
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30196: Human being; Person
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of training an object detection network, an object detection method and related apparatuses are provided. The training method includes: inputting a training image into a to-be-trained object detection network to obtain detection information of a target object, where the detection information includes a detection class of the target object, a detection position of a detection box of the target object, and a detection position of a landmark of the target object; calculating a total loss function of the to-be-trained object detection network, where the total loss function is calculated according to a loss function of the detection class of the target object, a loss function of the detection position of the detection box of the target object, and a loss function of the detection position of the landmark of the target object; and adjusting a parameter of the to-be-trained object detection network according to the total loss function.

## Claims

### Claim 1

A method of training an object detection network, comprising:
inputting a training image into a to-be-trained object detection network to obtain detection information of a target object in the training image, wherein the detection information comprises a detection class of the target object, a detection position of a detection box of the target object, and a detection position of a landmark of the target object inside the detection box;
calculating a total loss function of the to-be-trained object detection network, wherein the total loss function is calculated according to a first loss function of the detection class of the target object, a second loss function of the detection position of the detection box of the target object, and a third loss function of the detection position of the landmark of the target object inside the detection box; and
adjusting a parameter of the to-be-trained object detection network according to the total loss function of the to-be-trained object detection network, to obtain a trained object detection network.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the inputting the training image into the to-be-trained object detection network to obtain the detection information of the target object in the training image comprises:
inputting the training image into a first convolutional network of the to-be-trained object detection network, to obtain feature maps of a plurality of scales; and
individually inputting the feature maps of the scales into a second convolutional network of the to-be-trained object detection network, to obtain detection information at each pixel position in each of the feature maps,
wherein the to-be-trained object detection network comprises the first convolutional network and the second convolutional network.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the second convolutional network comprises a first convolutional layer with a convolution kernel size of 1*1*n, wherein the first convolutional layer is configured to convert a respective feature map into a first target feature map comprising the detection position of the landmark of the target object inside the detection box, and n is any positive integer.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the total loss function is calculated by using the following formula:
L=Lcls+α1Lbox+α2Lldm,
wherein L is the total loss function, Lcls is the first loss function, Lbox is the second loss function, Lldm is the third loss function, and α1 and α2 are weights.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the first loss function is calculated by using the following formula:
L
cls
=
-
∑
i
∈
Pos
N
x
ij
⁢
log
⁡
(
c
i
P
)
-
∑
i
∈
Neg
log
⁡
(
c
i
0
)
wherein i represents an ith candidate box, j represents a jth true box, p represents a pth class, xij represents whether the ith candidate box matches the jth true box, cip represents a probability that the ith candidate box is determined as the pth class, ci0 represents a probability that the ith candidate box is determined as a background class, Pos represents a set of candidate boxes that match the jth true box, and Neg represents a set of candidate boxes that do not match the jth true box.

### Claim 6 (depends on 4)

The method according to claim 4, wherein the second loss function is calculated by using the following formula:
L
box
=
∑
i
∈
Pos
N
x
i
⁢
j
⁢
smooth
L
⁢
1
(
l
bi
-
g
bj
)
wherein i represents an ith candidate box, j represents a jth true box, xij represents whether the ith candidate box matches the jth true box, Pos represents a set of candidate boxes that match the jth true box, smoothL1 represents a loss function, lbi represents a detection position of a detection box corresponding to the ith candidate box, and gbj is a position of the jth true box.

### Claim 7 (depends on 4)

The method according to claim 4, wherein the third loss function is calculated by using the following formula:
L
ldm
=
∑
i
∈
Pos
N
x
ij
⁢
smooth
L
⁢
1
(
l
li
-
g
lj
)
wherein i represents an ith candidate box, j represents a jth true box, xij represents whether the ith candidate box matches the jth true box, Pos represents a set of candidate boxes that match the jth true box, smoothL1 represents a loss function, lli represents a detection position of a landmark of a target object inside a detection box corresponding to the ith candidate box, and glj is a true position of a landmark of a target object in the jth true box.

### Claim 8 (depends on 1)

The method according to claim 1, wherein
the detection position of the detection box comprises: an offset amount of a horizontal coordinate of a center point of the detection box relative to a horizontal coordinate of a center point of a candidate box, an offset amount of a vertical coordinate of the center point of the detection box relative to a vertical coordinate of the center point of the candidate box, an offset amount of a length of the detection box relative to a length of the candidate box, and an offset amount of a width of the detection box relative to a width of the candidate box; and
the detection position of the landmark of the target object inside the detection box comprises a predicted horizontal coordinate of the landmark of the target object and a predicted vertical coordinate of the landmark of the target object.

### Claim 9 (depends on 1)

The method according to claim 1, wherein before the inputting the training image into the to-be-trained object detection network to obtain the detection information of the target object in the training image, the method further comprises:
randomly generating, for each to-be-enhanced training image, a color mask with a size the same as a size of the to-be-enhanced training image, wherein the color mask comprises only one color; and
fusing the to-be-enhanced training image and the color mask, to obtain a color-enhanced training image as the training image inputted into the to-be-trained object detection network.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the to-be-enhanced training image and the color mask are fused by using following formula:
imgaug=α*colormask+(1−α)*img,
wherein colormask represents the color mask, img represents the to-be-enhanced training image, α represents a weight, imgaug represents the color-enhanced training image, and 0<α<0.5.

### Claim 11 (depends on 1)

The method according to claim 1, wherein before the inputting the training image into the to-be-trained object detection network to obtain the detection information of the target object in the training image, the method further comprises:
converting, for each to-be-enhanced training image, the to-be-enhanced training image from an RGB color space into an HSV color space;
randomly transforming an H channel of the to-be-enhanced training image converted into the HSV color space, to obtain a transformed to-be-enhanced training image; and
converting the transformed to-be-enhanced training image back into the RGB color space, to obtain a color-enhanced training image as the training image inputted into the to-be-trained object detection network.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the target object is a hand, and the landmark is a point representing a joint position of the target object.

### Claim 13 (depends on 1)

An object detection method, comprising:
inputting a to-be-detected image into an object detection network, and outputting a detection position and a detection class of a detection box of a target object in the to-be-detected image,
wherein the object detection network is trained by using the method of training an object detection network according to claim 1.

### Claim 14 (depends on 13)

The object detection method according to claim 13, wherein the inputting the to-be-detected image into the object detection network, and outputting the detection position and the detection class of the detection box of the target object in the to-be-detected image comprises:
inputting the to-be-detected image into the object detection network, to obtain detection information of the target object in the to-be-detected image, wherein the detection information comprises the detection class of the target object, the detection position of the detection box of the target object, and a detection position of a landmark of the target object inside the detection box; and
outputting the detection position and the detection class of the detection box of the target object in the to-be-detected image based on the detection position of the landmark of the target object inside the detection box.

### Claim 15 (depends on 13)

The object detection method according to claim 13, wherein before the inputting the to-be-detected image into the object detection network, the method further comprises: obtaining a current to-be-detected image; and
the inputting the to-be-detected image into the object detection network, and outputting the detection position of the detection box of the target object in the to-be-detected image comprises:
recording the detection position of the detection box of the target object in a case that the object detection network detects the target object in a previous frame of inputted image;
determining a position of a crop box in the current to-be-detected image according to the detection position of the detection box of the target object in the previous frame of inputted image, wherein the detection box is within the crop box; and
cropping the current to-be-detected image based on the position of the crop box, to obtain the to-be-detected image to be inputted into the object detection network.

### Claim 16 (depends on 15)

The object detection method according to claim 15, wherein
coordinates of a center point of the crop box are the same as coordinates of a center point of the detection box, a length of the crop box is n times a length of a long side of the detection box, and a width of the crop box is m times the length of the long side of the detection box.

### Claim 17 (depends on 16)

The object detection method according to claim 16, wherein an aspect ratio of the to-be-detected image is n:m.

### Claim 18 (depends on 15)

The object detection method according to claim 15, further comprising:
inputting the current to-be-detected image into the object detection network as the to-be-detected image in a case that the object detection network fails to detect the target object in the previous frame of inputted image.

### Claim 19

An electronic device, comprising:
one or more memory; and
one or more processors;
wherein the one or more memory and the one or more processors are connected with each other; and
the one or more memory stores computer-executable instructions for controlling the one or more processors to:
input a training image into a to-be-trained object detection network to obtain detection information of a target object in the training image, wherein the detection information comprises a detection class of the target object, a detection position of a detection box of the target object, and a detection position of a landmark of the target object inside the detection box;
calculate a total loss function of the to-be-trained object detection network, wherein the total loss function is calculated according to a first loss function of the detection class of the target object, a second loss function of the detection position of the detection box of the target object, and a third loss function of the detection position of the landmark of the target object inside the detection box; and
adjust a parameter of the to-be-trained object detection network according to the total loss function of the to-be-trained object detection network, to obtain a trained object detection network.

### Claim 20

An electronic device, comprising:
one or more memory; and
one or more processors;
wherein the one or more memory and the one or more processors are connected with each other; and
the one or more memory stores computer-executable instructions for controlling the one or more processors to input a to-be-detected image into an object detection network, and outputting a detection position and a detection class of a detection box of a target object in the to-be-detected image;
wherein the one or more memory further stores computer-executable instructions for controlling the one or more processors to train the object detection network by:
inputting a training image into a to-be-trained object detection network to obtain detection information of a target object in the training image, wherein the detection information comprises a detection class of the target object, a detection position of a detection box of the target object, and a detection position of a landmark of the target object inside the detection box;
calculating a total loss function of the to-be-trained object detection network, wherein the total loss function is calculated according to a first loss function of the detection class of the target object, a second loss function of the detection position of the detection box of the target object, and a third loss function of the detection position of the landmark of the target object inside the detection box; and
adjusting a parameter of the to-be-trained object detection network according to the total loss function of the to-be-trained object detection network, to obtain a trained object detection network.