# US 11,907,848 B2 · Method and apparatus for training pose recognition model, and method and apparatus for image recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,848 B2
- Document ID: US11907848B2
- Status: Granted
- Kind: B2
- Application number: 17/330,261
- Filed: 2021-05-25
- Published: 2021-09-09
- Granted: 2024-02-20
- Earliest claimed: 2019-04-12
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Jingmin Luo; Xiaolong Zhu; Yitong Wang; Xing Ji
- Web page: https://patentgrep.com/patent/US11907848B2

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/74: {involving reference images or patches}
- G06T 7/75: {involving models}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06N 3/045: Combinations of networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

This application provides a method for training a pose recognition model performed at a computer device. The method includes: inputting a sample image labeled with human body key points into a feature map model included in a pose recognition model, to output a feature map of the sample image; inputting the feature map into a two-dimensional (2D) model included in the pose recognition model, to output 2D key point parameters used for representing a 2D human body pose; input a target human body feature map cropped from the feature map and the 2D key point parameter into a three-dimensional (3D) model included in the pose recognition model, to output 3D pose parameters used for representing a 3D human body pose; constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and updating the pose recognition model based on the target loss function.

## Claims

### Claim 1

A method for training a pose recognition model, applicable to a computer device, the method comprising:
processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image;
processing the feature map of the sample image by using a two-dimensional (2D) model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose;
processing, by using a three-dimensional (3D) model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose;
constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and
updating the pose recognition model based on the target loss function.

### Claim 2 (depends on 1)

The method according to claim 1, wherein before the processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image, the method further comprises:
determining the human body key points in a key point set according to a setting; and
labeling the sample image based on the human body key points and with reference to the key point set.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the key point set comprises:
reference key points used for localization of human body parts and extension key points, which, together with the reference key points, represent a plurality of 3D poses of the parts to which the reference key points and the extension key points belong.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the target loss function comprises a first loss function corresponding to the 3D model; and the constructing a target loss function based on the 2D key point parameters and the 3D pose parameters comprises:
determining corresponding 2D key point information based on the 3D pose parameters; and
constructing the first loss function based on the 2D key point parameters and the 2D key point information.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the target loss function further comprises a loss function corresponding to the 2D model and a second loss function corresponding to the 3D model; the 2D key point parameters comprise a part affinity field (PAF) parameter of the human body key points and a heatmap of the human body key points, and the 3D pose parameters comprise a shape parameter and a pose parameter of a human body; and
the constructing a target loss function based on the 2D key point parameters and the 3D pose parameters comprises:
constructing the loss function corresponding to the 2D model based on a difference between a PAF parameter outputted by the 2D model and the PAF parameter of the human body key points in the sample image and a difference between a heatmap outputted by the 2D model and the heatmap of the corresponding human body key points in the sample image; and
constructing the second loss function corresponding to the 3D model based on a difference between a shape parameter outputted by the 3D model and the shape parameter of the corresponding human body in the sample image and a difference between a pose parameter outputted by the 3D model and the pose parameter of the corresponding human body in the sample image.

### Claim 6 (depends on 1)

The method according to claim 1, wherein before the processing a target human body feature map cropped from the feature map and the 2D key point parameters by using a 3D model comprised in the pose recognition model, to obtain a 3D pose parameter used for representing a 3D human body pose, the method further comprises:
determining a target human body in the feature map based on the 2D key point parameters; and
cropping the feature map according to the target human body, to obtain the target human body feature map.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the updating a model parameter of the pose recognition model based on the target loss function comprises:
determining a value of the target loss function based on the 2D key point parameters and the 3D pose parameters;
determining an error signal of the pose recognition model based on the target loss function in a case that the value of the target loss function exceeds a preset threshold; and
back-propagating the error signal in the pose recognition model, and updating a model parameter of each layer during the propagation.

### Claim 8 (depends on 1)

The method according to claim 1, the method comprising:
processing, by using the feature map model comprised in the pose recognition model, a to-be-recognized image comprising a human body, to obtain a feature map of the to-be-recognized image;
processing the feature map of the to-be-recognized image by using the 2D model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose in the to-be-recognized image; and
processing, by using the 3D model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose in the to-be-recognized image.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising:
recognizing a 2D pose of the human body in the to-be-recognized image based on the 2D key point parameter, the to-be-recognized image being acquired based on an outputted image of a specific human pose;
performing similarity matching between the 2D pose and the specific human pose, to obtain a matching result; and
outputting prompt information used for representing the matching result.

### Claim 10 (depends on 8)

The method according to claim 8, further comprising:
constructing, based on the 3D pose parameter, a 3D human body model corresponding to the target human body; and
controlling the 3D human body model to perform a target action, the target action matching an action performed by the target human body.

### Claim 11

A computer device, comprising:
a memory, configured to store executable instructions; and
a processor, configured to perform, when executing the executable instructions stored in the memory, a plurality of operations including:
processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image;
processing the feature map of the sample image by using a two-dimensional (2D) model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose;
processing, by using a three-dimensional (3D) model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose;
constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and
updating the pose recognition model based on the target loss function.

### Claim 12 (depends on 11)

The computer device according to claim 11, wherein the plurality of operations further comprise:
before processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image:
determining the human body key points in a key point set according to a setting; and
labeling the sample image based on the human body key points and with reference to the key point set.

### Claim 13 (depends on 12)

The computer device according to claim 12, wherein the key point set comprises:
reference key points used for localization of human body parts and extension key points, which, together with the reference key points, represent a plurality of 3D poses of the parts to which the reference key points and the extension key points belong.

### Claim 14 (depends on 11)

The computer device according to claim 11, wherein the target loss function comprises a first loss function corresponding to the 3D model; and the constructing a target loss function based on the 2D key point parameters and the 3D pose parameters comprises:
determining corresponding 2D key point information based on the 3D pose parameters; and
constructing the first loss function based on the 2D key point parameters and the 2D key point information.

### Claim 15 (depends on 11)

The computer device according to claim 11, wherein the plurality of operations further comprise:
before processing a target human body feature map cropped from the feature map and the 2D key point parameters by using a 3D model comprised in the pose recognition model, to obtain a 3D pose parameter used for representing a 3D human body pose:
determining a target human body in the feature map based on the 2D key point parameters; and
cropping the feature map according to the target human body, to obtain the target human body feature map.

### Claim 16 (depends on 11)

The computer device according to claim 11, wherein the updating a model parameter of the pose recognition model based on the target loss function comprises:
determining a value of the target loss function based on the 2D key point parameters and the 3D pose parameters;
determining an error signal of the pose recognition model based on the target loss function in a case that the value of the target loss function exceeds a preset threshold; and
back-propagating the error signal in the pose recognition model, and updating a model parameter of each layer during the propagation.

### Claim 17 (depends on 11)

The computer device according to claim 11, wherein the plurality of operations further comprise:
processing, by using the feature map model comprised in the pose recognition model, a to-be-recognized image comprising a human body, to obtain a feature map of the to-be-recognized image;
processing the feature map of the to-be-recognized image by using the 2D model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose in the to-be-recognized image; and
processing, by using the 3D model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose in the to-be-recognized image.

### Claim 18

A non-transitory computer-readable storage medium, storing executable instructions that, when executed by a processor of a computer device, cause the computer device to perform a plurality of operations including:
processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image;
processing the feature map of the sample image by using a two-dimensional (2D) model comprised in the pose recognition model, to obtain 2D key point parameters used for representing a 2D human body pose;
processing, by using a three-dimensional (3D) model comprised in the pose recognition model, a target human body feature map cropped from the feature map and the 2D key point parameters, to obtain 3D pose parameters used for representing a 3D human body pose;
constructing a target loss function based on the 2D key point parameters and the 3D pose parameters; and
updating the pose recognition model based on the target loss function.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium according to claim 18, wherein the plurality of operations further comprise:
before processing, by using a feature map model comprised in a pose recognition model, a sample image labeled with human body key points, to obtain a feature map of the sample image:
determining the human body key points in a key point set according to a setting; and
labeling the sample image based on the human body key points and with reference to the key point set.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein the key point set comprises:
reference key points used for localization of human body parts and extension key points, which, together with the reference key points, represent a plurality of 3D poses of the parts to which the reference key points and the extension key points belong.