# US 2021/0174074 A1 · Human detection method and apparatus, computer device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174074 A1
- Document ID: US20210174074A1
- Status: Published application
- Kind: A1
- Application number: 17/181,376
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2019-09-27
- Applicants: Beijing Sensetime Technology Development Co., Ltd.
- Inventors: Haodong Duan; Wentao Liu
- Web page: https://patentgrep.com/patent/US20210174074A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/73: using feature-based methods
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A human detection method and apparatus, a computer device and a storage medium are provided. The method includes that: an image to be detected is acquired; position information of skeletal key points configured to represent a human skeletal structure and position information of contour key points configured to represent a human contour are determined based on the image to be detected; and a human detection result is generated based on the position information of the skeletal key points and the position information of the contour key points.

## Claims

### Claim 1

A human detection method, comprising:
acquiring an image to be detected;
determining, based on the image to be detected, position information of skeletal key points configured to represent a human skeletal structure and position information of contour key points configured to represent a human contour; and
generating a human detection result based on the position information of the skeletal key points and the position information of the contour key points.

### Claim 2 (depends on 1)

The human detection method of claim 1, wherein the contour key points comprises main contour key points and auxiliary contour key points, and there is at least one auxiliary contour key point between adjacent two of the main contour key points.

### Claim 3 (depends on 2)

The human detection method of claim 2, wherein determining, based on the image to be detected, the position information of the contour key points configured to represent the human contour comprises:
determining position information of the main contour key points based on the image to be detected;
determining human contour information based on the position information of the main contour key points; and
determining position information of multiple auxiliary contour key points based on the determined human contour information.

### Claim 4 (depends on 1)

The human detection method of claim 1, wherein the human detection result comprises at least one of:
the image to be detected added with skeletal key point tags and contour key point tags; or
a data set comprising the position information of the skeletal key points and the position information of the contour key points.

### Claim 5 (depends on 4)

The human detection method of claim 4, further comprising:
executing, based on the human detection result, at least one of the following operations: human action recognition, human pose detection, human contour regulation, human body image edition or human body mapping.

### Claim 6 (depends on 1)

The human detection method of claim 1, wherein determining, based on the image to be detected, the position information of the skeletal key points configured to represent the human skeletal structure and the position information of the contour key points configured to represent the human contour comprises:
performing, based on the image to be detected, feature extraction to obtain a skeletal feature and a contour feature, and performing feature fusion on the obtained skeletal feature and contour feature; and
determining, based on a feature fusion result, the position information of the skeletal key points and the position information of the contour key points.

### Claim 7 (depends on 6)

The human detection method of claim 6, wherein performing, based on the image to be detected, feature extraction to obtain the skeletal feature and the contour feature and performing feature fusion on the obtained skeletal feature and contour feature comprises:
performing, based on the image to be detected, at least one time of feature extraction, and performing feature fusion on a skeletal feature and contour feature obtained by each time of feature extraction, an (i+1)th time of feature extraction being performed based on a feature fusion result of an ith time of feature fusion under the condition that multiple feature extractions are performed, and i being a positive integer; and
determining, based on the feature fusion result, the position information of the skeletal key points configured to represent the human skeletal structure and the position information of the contour key points configured to represent the human contour comprises:
determining, based on a feature fusion result of a last time of feature fusion, the position information of the skeletal key points and the position information of the contour key points.

### Claim 8 (depends on 7)

The human detection method of claim 7, wherein performing, based on the image to be detected, at least one time of feature extraction comprises:
in a first time of feature extraction, extracting, from the image to be detected by use of a first feature extraction network which is pre-trained, a first target skeletal feature matrix of the skeletal key points configured to represent the skeletal feature and a first target contour feature matrix of the contour key points configured to represent the contour feature; and
in the (i+1)th time of feature extraction, extracting, from the feature fusion result of the ith time of feature fusion by use of a second feature extraction network which is pre-trained, the first target skeletal feature matrix and the first target contour feature matrix,
wherein network parameters of the first feature extraction network and the second feature extraction network being different, and network parameters of the second feature extraction network for different times of feature extraction being different.

### Claim 9 (depends on 8)

The human detection method of claim 8, wherein performing feature fusion on the obtained skeletal feature and contour feature comprises:
performing, by use of a feature fusion neural network which is pre-trained, feature fusion on the first target skeletal feature matrix and the first target contour feature matrix to obtain a second target skeletal feature matrix and a second target contour feature matrix,
wherein the second target skeletal feature matrix is a three-dimensional skeletal feature matrix, the three-dimensional skeletal feature matrix comprises two-dimensional skeletal feature matrices respectively corresponding to all skeletal key points, and a value of each element in the two-dimensional skeletal feature matrix represents a probability that a pixel corresponding to the element is the corresponding skeletal key point;
the second target contour feature matrix is a three-dimensional contour feature matrix, the three-dimensional contour feature matrix comprises two-dimensional contour feature matrices respectively corresponding to all contour key points, and a value of each element in the two-dimensional contour feature matrix represents a probability that a pixel corresponding to the element is the corresponding contour key point; and
network parameters of the feature fusion neural network for different times of feature fusion are different.

### Claim 10 (depends on 8)

The human detection method of claim 8, wherein the first feature extraction network comprises a common feature extraction network, a first skeletal feature extraction network and a first contour feature extraction network, and
extracting, from the image to be detected by use of the first feature extraction network, the first target skeletal feature matrix and the first target contour feature matrix comprises:
performing, by use of the common feature extraction network, convolution processing on the image to be detected to obtain a basic feature matrix comprising the skeletal feature and the contour feature;
performing, by use of the first skeletal feature extraction network, convolution processing on the basic feature matrix to obtain a first skeletal feature matrix;
acquiring a second skeletal feature matrix from a first target convolutional layer in the first skeletal feature extraction network;
obtaining the first target skeletal feature matrix based on the first skeletal feature matrix and the second skeletal feature matrix, the first target convolutional layer being any other convolutional layer, except a last convolutional layer, in the first skeletal feature extraction network;
performing, by use of the first contour feature extraction network, convolution processing on the basic feature matrix to obtain a first contour feature matrix;
acquiring a second contour feature matrix from a second target convolutional layer in the first contour feature extraction network; and
obtaining the first target contour feature matrix based on the first contour feature matrix and the second contour feature matrix, the second target convolutional layer being any other convolutional layer, except a last convolutional layer, in the first contour feature extraction network.

### Claim 11 (depends on 10)

The human detection method of claim 10, wherein obtaining the first target skeletal feature matrix based on the first skeletal feature matrix and the second skeletal feature matrix comprises:
performing concatenation processing on the first skeletal feature matrix and the second skeletal feature matrix to obtain a first concatenated skeletal feature matrix, and performing dimension transform processing on the first concatenated skeletal feature matrix to obtain the first target skeletal feature matrix; and
obtaining the first target contour feature matrix based on the first contour feature matrix and the second contour feature matrix comprises:
performing concatenation processing on the first contour feature matrix and the second contour feature matrix to obtain a first concatenated contour feature matrix, and performing dimension transform processing on the first concatenated contour feature matrix to obtain the first target contour feature matrix,
wherein a dimension of the first target skeletal feature matrix being the same as a dimension of the first target contour feature matrix, and the first target skeletal feature matrix and the first target contour feature matrix being the same in dimensionality in a same dimension.

### Claim 12 (depends on 9)

The human detection method of claim 9, wherein the feature fusion neural network comprises a first convolutional neural network, a second convolutional neural network, a first transform neural network and a second transform neural network, and
performing, by use of the feature fusion neural network, feature fusion on the first target skeletal feature matrix and the first target contour feature matrix to obtain the second target skeletal feature matrix and the second target contour feature matrix comprises:
performing, by use of the first convolutional neural network, convolution processing on the first target skeletal feature matrix to obtain a first intermediate skeletal feature matrix;
performing, by use of the second convolutional neural network, convolution processing on the first target contour feature matrix to obtain a first intermediate contour feature matrix;
performing concatenation processing on the first intermediate contour feature matrix and the first target skeletal feature matrix to obtain a first concatenated feature matrix;
performing, by use of the first transform neural network, dimension transform on the first concatenated feature matrix to obtain the second target skeletal feature matrix; and
performing concatenation processing on the first intermediate skeletal feature matrix and the first target contour feature matrix to obtain a second concatenated feature matrix; and
performing, by use of the second transform neural network, dimension transform on the second concatenated feature matrix to obtain the second target contour feature matrix.

### Claim 13 (depends on 9)

The human detection method of claim 9, wherein the feature fusion neural network comprises a first directional convolutional neural network, a second directional convolutional neural network, a third convolutional neural network, a fourth convolutional neural network, a third transform neural network and a fourth transform neural network, and
performing, by use of the feature fusion neural network, feature fusion on the first target skeletal feature matrix and the first target contour feature matrix to obtain the second target skeletal feature matrix and the second target contour feature matrix comprises:
performing, by use of the first directional convolutional neural network, directional convolution processing on the first target skeletal feature matrix to obtain a first directional skeletal feature matrix;
performing, by use of the third convolutional neural network, convolution processing on the first directional skeletal feature matrix to obtain a second intermediate skeletal feature matrix;
performing, by use of the second directional convolutional neural network, directional convolution processing on the first target contour feature matrix to obtain a first directional contour feature matrix;
performing, by use of the fourth convolutional neural network, convolution processing on the first directional contour feature matrix to obtain a second intermediate contour feature matrix;
performing concatenation processing on the second intermediate contour feature matrix and the first target skeletal feature matrix to obtain a third concatenated feature matrix;
performing, by use of the third transform neural network, dimension transform on the third concatenated feature matrix to obtain the second target skeletal feature matrix;
performing concatenation processing on the second intermediate skeletal feature matrix and the first target contour feature matrix to obtain a fourth concatenated feature matrix, and
performing, by use of the fourth transform neural network, dimension transform on the fourth concatenated feature matrix to obtain the second target contour feature matrix.

### Claim 14 (depends on 9)

The human detection method of claim 9, wherein the feature fusion neural network comprises a shift estimation neural network and a fifth transform neural network, and
performing, by use of the feature fusion neural network, feature fusion on the first target skeletal feature matrix and the first target contour feature matrix to obtain the second target skeletal feature matrix and the second target contour feature matrix comprises:
performing concatenation processing on the first target skeletal feature matrix and the first target contour feature matrix to obtain a fifth concatenated feature matrix;
inputting the fifth concatenated feature matrix to the shift estimation neural network, and performing shift estimation on multiple predetermined key point pairs to obtain shift information of a shift from one key point in each key point pair to the other key point in the key point pair;
by taking each key point in each key point pair as a present key point respectively, acquiring, from a three-dimensional feature matrix corresponding to the other key point paired with the present key point, a two-dimensional feature matrix corresponding to the paired other key point;
performing, according to shift information of a shift from the paired other key point to the present key point, positional shifting on elements in the two-dimensional feature matrix corresponding to the paired other key point to obtain a shift feature matrix corresponding to the present key point;
for each skeletal key point, performing concatenation processing on a two-dimensional feature matrix corresponding to the skeletal key point and each shift feature matrix corresponding to the skeletal key point to obtain a concatenated two-dimensional feature matrix of the skeletal key point;
inputting the concatenated two-dimensional feature matrix of the skeletal key point to the fifth transform neural network to obtain a target two-dimensional feature matrix corresponding to the skeletal key point;
generating the second target skeletal feature matrix based on the target two-dimensional feature matrices respectively corresponding to all skeletal key points;
for each contour key point, performing concatenation processing on a two-dimensional feature matrix corresponding to the contour key point and each shift feature matrix corresponding to the contour key point to obtain a concatenated two-dimensional feature matrix of the contour key point;
inputting the concatenated two-dimensional feature matrix of the contour key point to the fifth transform neural network to obtain a target two-dimensional feature matrix corresponding to the contour key point; and
generating the second target contour feature matrix based on the target two-dimensional feature matrices respectively corresponding to all contour key points.

### Claim 15 (depends on 1)

The human detection method of claim 1, wherein the human detection method is implemented through a human detection model, and the human detection model comprises a first feature extraction network and/or a feature fusion neural network; and
wherein the human detection model is obtained by training through sample images in a training sample set, the sample images being tagged with practical position information of the skeletal key points of the human skeletal structure and practical position information of the contour key points of the human contour.

### Claim 16

A computer device, comprising a processor, a non-transitory storage medium and a bus, wherein the non-transitory storage medium stores machine-readable instructions executable for the processor; under the condition that the computer device runs, the processor communicates with the non-transitory storage medium through the bus; and the machine-readable instructions are executed by the processor, which caused that the processor is configured to:
acquire an image to be detected;
determine, based on the image to be detected, position information of skeletal key points configured to represent a human skeletal structure and position information of contour key points configured to represent a human contour; and
generate a human detection result based on the position information of the skeletal key points and the position information of the contour key points.

### Claim 17 (depends on 16)

The computer device of claim 16, wherein the processor is configured to determine, based on the image to be detected, the position information of the skeletal key points configured to represent the human skeletal structure and the position information of the contour key points configured to represent the human contour in the following manner:
performing, based on the image to be detected, feature extraction to obtain a skeletal feature and a contour feature, and performing feature fusion on the obtained skeletal feature and contour feature; and
determining, based on a feature fusion result, the position information of the skeletal key points and the position information of the contour key points.

### Claim 18 (depends on 17)

The computer device of claim 17, wherein the processor is configured to perform, based on the image to be detected, feature extraction to obtain the skeletal feature and the contour feature and perform feature fusion on the obtained skeletal feature and contour feature in the following manner:
performing, based on the image to be detected, at least one time of feature extraction, and performing feature fusion on a skeletal feature and contour feature obtained by each time of feature extraction, an (i+1)th time of feature extraction being performed based on a feature fusion result of an ith time of feature fusion under the condition that multiple feature extractions are performed, and i being a positive integer; and
the processor is configured to is configured to determine, based on the feature fusion result, the position information of the skeletal key points configured to represent the human skeletal structure and the position information of the contour key points configured to represent the human contour in the following manner:
determining, based on a feature fusion result of a last time of feature fusion, the position information of the skeletal key points and the position information of the contour key points.

### Claim 19 (depends on 18)

The computer device of claim 18, wherein the processor is configured to perform, based on the image to be detected, at least one time of feature extraction in the following manner:
in a first time of feature extraction, extracting, from the image to be detected by use of a first feature extraction network which is pre-trained, a first target skeletal feature matrix of the skeletal key points configured to represent the human skeletal structure and a first target contour feature matrix of the contour key points configured to represent the human contour; and
in the (i+1)th time of feature extraction, extracting, from the feature fusion result of the ith time of feature fusion by use of a second feature extraction network which is pre-trained, the first target skeletal feature matrix and the first target contour feature matrix,
wherein network parameters of the first feature extraction network and the second feature extraction network being different, and network parameters of the second feature extraction network for different times of feature extraction being different.

### Claim 20

A non-transitory computer-readable storage medium, in which computer programs is stored, wherein the computer programs are operated by a processor to execute:
acquiring an image to be detected;
determining, based on the image to be detected, position information of skeletal key points configured to represent a human skeletal structure and position information of contour key points configured to represent a human contour; and
generating a human detection result based on the position information of the skeletal key points and the position information of the contour key points.
