# US 2022/0044056 A1 · Method and apparatus for detecting keypoints of human body, electronic device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044056 A1
- Document ID: US20220044056A1
- Status: Published application
- Kind: A1
- Application number: 17/507,850
- Filed: 2021-10-22
- Published: 2022-02-10
- Earliest claimed: 2019-07-15
- Assignees: Shenzhen Sensetime Technology Co., Ltd.
- Applicants: Shenzhen Sensetime Technology Co., Ltd.
- Inventors: Wentao LIU; Yujing GUO; Yongwang WANG; Chen QIAN; Jiahua LI
- Web page: https://patentgrep.com/patent/US20220044056A1

## CPC

- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06F 18/25: Fusion techniques
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/82: using neural networks
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method and an apparatus for detecting key points of a human body, and a storage medium are provided. The method includes: extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions.

## Claims

### Claim 1

A method for detecting body keypoints, comprising;
extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and
performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions.

### Claim 2 (depends on 1)

The method of claim 1, wherein before performing the body keypoint feature fusion on the 2D pose data and the depth data corresponding to the body keypoint positions, the method further comprises:
performing data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the frame of image to obtain an RGB depth (RGBD) image data stream.

### Claim 3 (depends on 1)

The method of claim 1, wherein detecting that the image comprises the body comprises:
obtaining multiple image features by performing first image processing on a present frame of image; and
in response to determining that the multiple image features are body keypoint features by using a body identification network, detecting that the present frame of image comprises the body till detection on at least one frame of image is completed.

### Claim 4 (depends on 2)

The method of claim 2, wherein before performing body keypoint feature fusion on the 2D pose data and the depth data that corresponds to the body keypoint positions, the method further comprises:
obtaining multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
acquiring a first body motion state;
describing, through first 3D pose data, a change of body keypoint positions corresponding to the first body motion state; and
generating a first control instruction according to the first 3D pose data, and sending the first control instruction to a receiving side device, so as to display an action simulating operation corresponding to the first body motion state on a display screen of the receiving side device.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
acquiring a second body motion state;
describing, through second 3D pose data, a change of body keypoint positions corresponding to the second body motion state;
comparing the second 3D pose data with pre-configured pose data, and generating a second control instruction in response to an inconsistent comparison result; and
sending prompt information according to the second control instruction, so as to adjust the second body motion state to a target state according to the prompt information.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
acquiring a third body motion state;
describing, through third 3D pose data, a change of body keypoint positions corresponding to the third body motion state; and
sending the third 3D pose data to a receiving side device, so as to display, on a display screen of the receiving side device, operations that a virtual avatar sampling the third 3D pose data.

### Claim 8 (depends on 3)

The method of claim 3, wherein a training process of the body identification network comprises:
taking pre-labeled body keypoint features as training sample data, inputting the training sample data to a to-be-trained body identification network for training till an output result meets network training conditions, and obtaining a trained body identification network.

### Claim 9

An apparatus for detecting body keypoints, comprising;
a processor; and
a memory configured to store instructions executable by the processor,
wherein the processor is configured to:
extract, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and
perform body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the processor is further configured to:
perform data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the each frame of image to obtain an RGB depth (RGBD) image data stream.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the processor is further configured to:
obtain multiple image features by performing first image processing on a present frame of image; and
in response to determining that the multiple image features are body keypoint features by using a body identification network, detect that the present frame of image comprises the body until detection on at least one frame of image is completed.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the processor is further configured to:
obtain multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed.

### Claim 13 (depends on 9)

The apparatus of claim 9, wherein the processor is further configured to:
acquire a first body motion state;
describe, through first 3D pose data, a change of body keypoint positions corresponding to the first body motion state; and
generate a first control instruction according to the first 3D pose data, and send the first control instruction to a receiving side device, so as to display an action simulating operation corresponding to the first body motion state on a display screen of the receiving side device.

### Claim 14 (depends on 9)

The apparatus of claim 9, wherein the processor is further configured to:
acquire a second body motion state;
describe, through second 3D pose data, a change of body keypoint positions corresponding to the second body motion state;
compare the second 3D pose data with pre-configured pose data, and generate a second control instruction in response to an inconsistent comparison result; and
send prompt information according to the second control instruction, so as to adjust the second body motion state to a target state according to the prompt information.

### Claim 15 (depends on 9)

The apparatus of claim 9, wherein the processor is further configured to:
acquire a third body motion state;
describe, through third 3D pose data, a change of body keypoint positions corresponding to the third body motion state; and
send the third 3D pose data to a receiving side device, so as to display, on a display screen of the receiving side device, operations that a virtual avatar sampling the third 3D pose data.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein the processor is further configured to:
take pre-labeled body keypoint features as training sample data in a training process of the body identification network, input the training sample data to a to-be-trained body identification network for training till an output result meets a network training condition, and obtain a trained body identification network.

### Claim 17

A non-transitory computer-readable storage medium, having computer program instructions stored thereon, wherein the computer program instructions, when being executed by a processor, cause to implement operations, the operations comprising:
extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and
performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein before performing the body keypoint feature fusion on the 2D pose data and the depth data corresponding to the body keypoint positions, the operations further comprise:
performing data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the frame of image to obtain an RGB depth (RGBD) image data stream.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the operation of detecting that the image comprises the body comprises:
obtaining multiple image features by performing first image processing on a present frame of image; and
in response to determining that the multiple image features are body keypoint features by using a body identification network, detecting that the present frame of image comprises the body till detection on at least one frame of image is completed.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein before performing body keypoint feature fusion on the 2D pose data and the depth data that corresponds to the body keypoint positions, the operations further comprise:
obtaining multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed.
