# US 2021/0241015 A1 · Image processing method and apparatus, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241015 A1
- Document ID: US20210241015A1
- Status: Published application
- Kind: A1
- Application number: 16/998,285
- Filed: 2020-08-20
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: Beijing Sensetime Technology Development Co., Ltd.
- Applicants: Beijing Sensetime Technology Development Co., Ltd.
- Inventors: Guanglu SONG; Yu LIU
- Web page: https://patentgrep.com/patent/US20210241015A1

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/107: {Static hand or arm}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure relates to an image processing method and apparatus, an electronic device, and a storage medium. The method includes: performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image; performing target region prediction on the first feature map to determine a first region where a target is located in the first feature map; and performing key point detection on the first feature map according to the first region to determine target key point information of the to-be-processed image. According to embodiments of the present disclosure, quick and accurate target key point detection can be implemented.

## Claims

### Claim 1

An image processing method, comprising:
performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image;
performing target region prediction on the first feature map to determine a first region where a target is located in the first feature map; and
performing key point detection on the first feature map according to the first region to determine target key point information of the to-be-processed image.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
performing bounding box regression on the target key point information of each target in the to-be-processed image respectively to determine a second region of the each target; and
determining a target recognition result of the to-be-processed image according to the second region of the each target.

### Claim 3 (depends on 1)

The method according to claim 1, wherein performing target region prediction on the first feature map to determine the first region where the target is located in the first feature map comprises:
performing center point feature extraction on the first feature map to obtain a second feature map;
determining at least one region center point from a plurality of feature points of the second feature map, wherein the region center point comprises a feature point having a feature value greater than or equal to a first threshold;
determining a scale of the first region corresponding to each region center point according to a channel number corresponding to the each region center point in the at least one region center point; and
determining each first region respectively according to a position of the each region center point and the scale of the first region corresponding to the each region center point.

### Claim 4 (depends on 1)

The method according to claim 1, wherein performing key point detection on the first feature map according to the first region to determine the target key point information of the to-be-processed image comprises:
performing key point feature extraction on the first feature map to obtain a third feature map;
determining a plurality of key points of the target from a feature region corresponding to the first region in the third feature map; and
determining the target key point information of the to-be-processed image according to positions of the plurality of key points in the third feature map.

### Claim 5 (depends on 4)

The method according to claim 4, wherein determining the plurality of key points of the target from the feature region corresponding to the first region in the third feature map comprises:
performing key point detection on each channel of the feature region respectively to obtain key points corresponding to the each channel.

### Claim 6 (depends on 2)

The method according to claim 2, wherein determining the target recognition result of the to-be-processed image according to the second region of the each target comprises:
determining a first confidence score of the first region corresponding to the second region, and a second confidence score of all or part of the target key point information of the second region;
determining a third confidence score of the second region according to the first confidence score and the second confidence score; and
determining the target recognition result of the to-be-processed image according to the third confidence score of each second region, wherein the target recognition result comprises the second region having the third confidence score greater than or equal to a second threshold.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the method is implemented via a neural network, and the method further comprises:
training the neural network according to a preset training set, wherein the training set comprises a plurality of sample images, region labeling information in the plurality of sample images, and key point labeling information in the plurality of sample images.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the neural network comprises a feature extraction network, a region prediction network, and a key point detection network, and
training the neural network according to the preset training set comprises:
processing the sample images via the feature extraction network to determine a first sample feature map;
processing the first sample feature map via the region prediction network to determine a second sample feature map and at least one sample region in the second sample feature map;
processing the first sample feature map via the key point detection network to determine a third sample feature map, wherein a width and a height of the third sample feature map are identical to a width and a height of the second sample feature map;
determining sample key point information in the sample region of the third sample feature map according to the at least one sample region;
determining a first labeling feature map of the sample images according to the region labeling information of the sample images, wherein a scale of the first labeling feature map are identical to a scale of the second sample feature map, and a channel number of the first labeling feature map represents a scale of the region labeling information;
determining a first loss of the sample images for the region prediction network according to a difference between the first labeling feature map and the second sample feature map;
determining a second loss of the sample images for the key point detection network according to a difference between the key point labeling information and the sample key point information; and
training the neural network according to the first loss and the second loss of the plurality of sample images.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the target in the to-be-processed image comprises any one of a face, a body, or hands.

### Claim 10

An image processing apparatus, comprising:
a processor; and
a memory configured to store processor-executable instructions,
wherein the processor is configured to invoke the instructions stored in the memory, so as to:
perform feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image;
perform target region prediction on the first feature map to determine a first region where a target is located in the first feature map; and
perform key point detection on the first feature map according to the first region to determine target key point information of the to-be-processed image.

### Claim 11 (depends on 10)

The apparatus according to claim 10, wherein the processor is further configured to:
perform bounding box regression on the target key point information of each target in the to-be-processed image respectively to determine a second region of the each target; and
determine a target recognition result of the to-be-processed image according to the second region of the each target.

### Claim 12 (depends on 10)

The apparatus according to claim 10, wherein performing target region prediction on the first feature map to determine the first region where the target is located in the first feature map comprises:
performing center point feature extraction on the first feature map to obtain a second feature map;
determining at least one region center point from a plurality of feature points of the second feature map, wherein the region center point comprises a feature point having a feature value greater than or equal to a first threshold;
determining a scale of the first region corresponding to each region center point according to a channel number corresponding to the each region center point in the at least one region center point; and
determining each first region respectively according to a position of the each region center point and the scale of the first region corresponding to the each region center point.

### Claim 13 (depends on 10)

The apparatus according to claim 10, wherein performing key point detection on the first feature map according to the first region to determine the target key point information of the to-be-processed image comprises:
performing key point feature extraction on the first feature map to obtain a third feature map;
determining a plurality of key points of the target from a feature region corresponding to the first region in the third feature map; and
determining the target key point information of the to-be-processed image according to positions of the plurality of key points in the third feature map.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein determining the plurality of key points of the target from the feature region corresponding to the first region in the third feature map comprises:
performing key point detection on each channel of the feature region respectively to obtain key points corresponding to the each channel.

### Claim 15 (depends on 11)

The apparatus according to claim 11, wherein determining the target recognition result of the to-be-processed image according to the second region of the each target comprises:
determining a first confidence score of the first region corresponding to the second region, and a second confidence score of all or part of the target key point information of the second region;
determining a third confidence score of the second region according to the first confidence score and the second confidence score; and
determining the target recognition result of the to-be-processed image according to the third confidence score of each second region, wherein the target recognition result comprises the second region having the third confidence score greater than or equal to a second threshold.

### Claim 16 (depends on 10)

The apparatus according to claim 10, wherein the processor is further configured to:
train a neural network according to a preset training set, wherein the training set comprises a plurality of sample images, region labeling information in the plurality of sample images, and key point labeling information in the plurality of sample images.

### Claim 17 (depends on 16)

The apparatus according to claim 16, wherein the neural network comprises a feature extraction network, a region prediction network, and a key point detection network, and
training the neural network according to the preset training set comprises:
processing the sample images via the feature extraction network to determine a first sample feature map;
processing the first sample feature map via the region prediction network to determine a second sample feature map and at least one sample region in the second sample feature map;
processing the first sample feature map via the key point detection network to determine a third sample feature map, wherein a width and a height of the third sample feature map are identical to a width and a height of the second sample feature map;
determining sample key point information in the sample region of the third sample feature map according to the at least one sample region;
determining a first labeling feature map of the sample images according to the region labeling information of the sample images, wherein a scale of the first labeling feature map are identical to a scale of the second sample feature map, and a channel number of the first labeling feature map represents a scale of the region labeling information;
determining a first loss of the sample images for the region prediction network according to a difference between the first labeling feature map and the second sample feature map;
determining a second loss of the sample images for the key point detection network according to a difference between the key point labeling information and the sample key point information; and
training the neural network according to the first loss and the second loss of the plurality of sample images.

### Claim 18 (depends on 10)

The apparatus according to claim 10, wherein the target in the to-be-processed image comprises any one of a face, a body, or hands.

### Claim 19

A non-transitory computer-readable storage medium, having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to perform the operations of:
performing feature extraction on a to-be-processed image to obtain a first feature map of the to-be-processed image;
performing target region prediction on the first feature map to determine a first region where a target is located in the first feature map; and
performing key point detection on the first feature map according to the first region to determine target key point information of the to-be-processed image.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein when the computer program instructions are executed by a processor, the processor is further caused to perform the operations of:
performing bounding box regression on the target key point information of each target in the to-be-processed image respectively to determine a second region of the each target; and
determining a target recognition result of the to-be-processed image according to the second region of the each target.