# US 12,223,693 B2 · Object detection method, object detection apparatus, and object detection system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,693 B2
- Document ID: US12223693B2
- Status: Granted
- Kind: B2
- Application number: 17/772,923
- Filed: 2021-04-01
- Published: 2024-05-16
- Granted: 2025-02-11
- Earliest claimed: 2021-04-01
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Chunshan Zu
- Web page: https://patentgrep.com/patent/US12223693B2

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/73: using feature-based methods
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06V 2201/07: Target detection
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An object detection method includes: obtaining a video to be detected; preprocessing the video to be detected to obtain an image to be detected; inputting the image to be detected into an object detection network; extracting, by the object detection network, a feature map of the image to be detected; performing, by the object detection network, an object prediction on the extracted feature map to obtain a position of an object in the image to be detected and a confidence degree corresponding to the position; and generating a marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected.

## Claims

### Claim 1

An object detection method, comprising:
obtaining a video to be detected;
preprocessing the video to be detected to obtain an image to be detected;
inputting the image to be detected into an object detection network;
extracting, by the object detection network, a feature map from the image to be detected;
performing, by the object detection network, an object prediction on the extracted feature map, so as to obtain a position of an object in the image to be detected and a confidence degree corresponding to the position, wherein the object detection network includes multi-layer cascade networks, a feature map extracted by a cascade network in each layer is obtained according to a first feature map and a second feature map, the first feature map is obtained by performing a convolution on a feature map extracted by a previous-layer cascade network, and the second feature map is obtained by performing a linear transformation on the first feature map; and
generating a marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected;
wherein the object detection network includes a feature extraction network, a feature fusion network and a prediction network; the feature extraction network includes the multi-layer cascade networks; and
inputting the image to be detected into the object detection network, extracting, by the object detection network, the feature map from the image to be detected, and performing, by the object detection network, the object prediction on the extracted feature map, so as to obtain the position of the object in the image to be detected and the confidence degree corresponding to the position, includes:
inputting the image to be detected into the feature extraction network;
performing, by the cascade network in each layer, the convolution on the feature map extracted by the previous-layer cascade network to obtain the first feature map;
performing, by the cascade network in each layer, the linear transformation on the first feature map to obtain the second feature map;
obtaining, by the cascade network in each layer, the feature map extracted by the cascade network in each layer according to the first feature map and the second feature map;
inputting a plurality of feature maps extracted by the multi-layer cascade networks into the feature fusion network to fuse the feature maps, so as to obtain fused feature maps; and
inputting the fused feature maps into the prediction network for object prediction, so as to obtain the position of the object in the image to be detected and the confidence degree corresponding to the position.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the cascade network in each layer includes a feature map extraction network and a weight learning network; and
performing, by the cascade network in each layer, the convolution on the feature map extracted by the previous-layer cascade network to obtain the first feature map, performing, by the cascade network in each layer, the linear transformation on the first feature map to obtain the second feature map, and obtaining, by the cascade network in each layer, the feature map extracted by the cascade network in each layer according to the first feature map and the second feature map, includes:
performing, by the feature map extraction network, a convolution on the feature map extracted by the previous-layer cascade network to obtain the first feature map;
performing, by the feature map extraction network, a linear transformation on the first feature map to obtain the second feature map;
combining, by the feature map extraction network, the first feature map and the second feature map to generate a combined feature map; and
learning, by the weight learning network, weight of each channel feature in the combined feature map to obtain the feature map extracted by the cascade network in each layer.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the feature map extraction network includes at least one group of neural network modules; each neural network module is used to perform a convolution, a linear transformation and a combination on an input content of the neural network module, so as to generate a combined feature map corresponding to the neural network module; and
the method further comprises:
obtaining sample images and positions of objects in the sample images;
preprocessing the sample images to obtain sample images to be detected;
training an initial object detection network by using the sample images to be detected and the positions of the objects in the sample images, so as to determine a number of groups of neural network modules included in the feature map extraction network in the cascade network in each layer; and
generating the object detection network according to the determined number of groups of neural network modules.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the feature extraction network includes three-layer cascade networks, the three-layer cascade networks including a first-layer cascade network, a second-layer cascade network, and a third-layer cascade network; and
a feature map extraction network in the first-layer cascade network includes one group of neural network modules, a feature map extraction network in the second-layer cascade network includes nine groups of neural network modules, and a feature map extraction network in the third-layer cascade network includes nine groups of neural network modules.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the feature map extraction network further includes a network layer including a depthwise separable convolution and at least one neural network module.

### Claim 6 (depends on 5)

The method according to claim 5, wherein at least one output of the at least one group of neural network modules in the feature map extraction network is connected to an input of the network layer;
the network layer includes a first neural network module, the depthwise separable convolution, a second neural network module, and a summation operation; and
the input of the network layer is an input of the first neural network module, an output of the first neural network module is connected to an input of the depthwise separable convolution, an output of the depthwise separable convolution is connected to an input of the second neural network module, an output of the second neural network module and the input of the network layer are connected to an input of the summation operation, and an output of the summation operation is an output of the network layer.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the object detection network obtains positions of objects in the image to be detected and confidence degrees corresponding to the positions; and generating the marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected, includes:
performing a coordinate transformation on a target position in the positions of the objects in the image to be detected according to the confidence degrees corresponding to the positions and an image size corresponding to the video to be detected, so as to obtain a transformed position corresponding to the image to be detected; and
marking an object in an original image corresponding to the image to be detected in the video to be detected according to the transformed position corresponding to the image to be detected, so as to generate the marked object video; wherein
a confidence degree corresponding to the target position is greater than confidence degrees corresponding to other positions, and the other positions are positions other than the target position in the positions of the objects in the image to be detected; and the transformed position corresponding to the image to be detected represents a position of a detection frame of the object in the image to be detected in the original image corresponding to the image to be detected in the video to be detected.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the image to be detected includes an i-th frame of image, a j-th frame of image, a g-th frame of image, and an h-th frame of image; i, j, g, and h are positive integers, i is greater than j, j is greater than g, and g is greater than h; and
while preprocessing the video to be detected to obtain the i-th frame of image, the object detection network extracts a feature map from the j-th frame of image, performs an object prediction on the extracted feature map, and performs a coordinate transformation on a target position in positions of objects in the g-th frame of image, and marks an object in an original image corresponding to the h-th frame of image in the video to be detected.

### Claim 9 (depends on 1)

An object detection apparatus comprising a memory and a processor; wherein the memory is coupled to the processor; the memory is used to store computer program codes, and the computer program codes includes computer instructions; and
the computer instructions, when executed by the processor, cause the apparatus to perform the object detection method according to claim 1.

### Claim 10 (depends on 9)

An object detection system comprising the object detection apparatus according to claim 9, and at least one camera, wherein the object detection apparatus is connected to the at least one camera, and the at least one camera is used to acquire the video to be detected.

### Claim 11 (depends on 1)

A non-transitory computer-readable storage medium storing computer programs that, when run on an object detection apparatus, cause the object detection apparatus to perform the object detection method according to claim 1.

### Claim 12 (depends on 4)

The method according to claim 4, wherein the feature map extraction network further includes a network layer including a depthwise separable convolution and at least one neural network module.

### Claim 13 (depends on 12)

The method according to claim 12, wherein at least one output of the at least one group of neural network modules in the feature map extraction network is connected to an input of the network layer;
the network layer includes a first neural network module, the depthwise separable convolution, a second neural network module, and a summation operation; and
the input of the network layer is an input of the first neural network module, an output of the first neural network module is connected to an input of the depthwise separable convolution, an output of the depthwise separable convolution is connected to an input of the second neural network module, an output of the second neural network module and the input of the network layer are connected to an input of the summation operation, and an output of the summation operation is an output of the network layer.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the video to be detected includes video streams; and preprocessing the video to be detected to obtain the image to be detected, includes:
preprocessing each video stream in the video to be detected to obtain a plurality of frames of images, so as to obtain the image to be detected, wherein the plurality of frames of images constitute an image to be detected corresponding to the video stream, and the image to be detected corresponding to the video to be detected includes the plurality of frames of images in the image to be detected corresponding to the video stream.

### Claim 15 (depends on 14)

The method according to claim 14, wherein extracting, by the object detection network, the feature map from the image to be detected, and performing, by the object detection network, the object prediction on the extracted feature map to obtain the position of the object in the image to be detected and the confidence degree corresponding to the position, includes:
extracting, by the object detection network, feature maps from the plurality of frames of images in the image to be detected corresponding to the video stream; and
performing, by the object detection network, the object prediction on the extracted feature maps to obtain positions of objects in the plurality of frames of images in the image to be detected corresponding to the video stream and confidence degrees corresponding to the positions.

### Claim 16 (depends on 15)

The method according to claim 15, wherein generating the marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected, includes:
generating a marked object video corresponding to the video stream according to the positions of the objects in the plurality of frames of images in the image to be detected corresponding to the video stream, the confidence degrees corresponding to the positions, and the video stream, so as to obtain marked object videos corresponding to the video streams in the video to be detected.

### Claim 17

An object detection method, comprising:
obtaining a video to be detected;
preprocessing the video to be detected to obtain an image to be detected;
inputting the image to be detected into an object detection network;
extracting, by the object detection network, a feature map from the image to be detected;
performing, by the object detection network, an object prediction on the extracted feature map, so as to obtain a position of an object in the image to be detected and a confidence degree corresponding to the position, wherein the object detection network includes multi-layer cascade networks, a feature map extracted by a cascade network in each layer is obtained according to a first feature map and a second feature map, the first feature map is obtained by performing a convolution on a feature map extracted by a previous-layer cascade network, and the second feature map is obtained by performing a linear transformation on the first feature map; and
generating a marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected;
wherein the object detection network obtains positions of objects in the image to be detected and confidence degrees corresponding to the positions; and generating the marked object video according to the position of the object in the image to be detected, the confidence degree corresponding to the position, and the video to be detected, includes:
performing a coordinate transformation on a target position in the positions of the objects in the image to be detected according to the confidence degrees corresponding to the positions and an image size corresponding to the video to be detected, so as to obtain a transformed position corresponding to the image to be detected; and
marking an object in an original image corresponding to the image to be detected in the video to be detected according to the transformed position corresponding to the image to be detected, so as to generate the marked object video; wherein
a confidence degree corresponding to the target position is greater than confidence degrees corresponding to other positions, and the other positions are positions other than the target position in the positions of the objects in the image to be detected; and the transformed position corresponding to the image to be detected represents a position of a detection frame of the object in the image to be detected in the original image corresponding to the image to be detected in the video to be detected;
the image to be detected includes an i-th frame of image, a j-th frame of image, a g-th frame of image, and an h-th frame of image; i, j, g, and h are positive integers, i is greater than j, j is greater than g, and g is greater than h; and
while preprocessing the video to be detected to obtain the i-th frame of image, the object detection network extracts a feature map from the j-th frame of image, performs an object prediction on the extracted feature map, and performs a coordinate transformation on a target position in positions of objects in the g-th frame of image, and marks an object in an original image corresponding to the h-th frame of image in the video to be detected.