# US 2021/0240971 A1 · Data processing method and apparatus, electronic device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240971 A1
- Document ID: US20210240971A1
- Status: Published application
- Kind: A1
- Application number: 17/049,687
- Filed: 2019-04-23
- Published: 2021-08-05
- Earliest claimed: 2018-09-18
- Assignees: Beijing Sensetime Technology Development Co., Ltd.
- Applicants: Beijing Sensetime Technology Development Co., Ltd.
- Inventors: Min WANG; Zhuang ZOU; Wentao LIU; Chen QIAN; Lizhuang MA
- Web page: https://patentgrep.com/patent/US20210240971A1

## CPC

- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 2207/30196: Human being; Person
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided are a data processing method and apparatus, an electronic device and a storage medium. The data processing method comprises: according to a reference depth value and an actual depth value of a key point of a target in an image, converting first 2D coordinates of the key point into second 2D coordinates, wherein the second 2D coordinates and the reference depth value constitute a first 3D feature of the key point; and based on the first 3D feature, obtaining a 3D posture of the target.

## Claims

### Claim 1

A data processing method, comprising:
converting, according to actual depth values of key points of a target in an image and reference depth values, first two-dimensional (2D) coordinates of the key points into second 2D coordinates, wherein the second 2D coordinates and the reference depth values form first 3D features of the key points; and
obtaining, based on the first 3D features, a 3D posture of the target.

### Claim 2 (depends on 1)

The method of claim 1, wherein
converting, according to the actual depth values of the key points of the target in the image and the reference depth values, the first 2D coordinates of the key points into the second 2D coordinates comprises:
obtaining the second 2D coordinates according to the first 2D coordinates and ratios of the actual depth values to the reference depth values.

### Claim 3 (depends on 2)

The method of claim 2, wherein
obtaining the second 2D coordinates according to the first 2D coordinates and the ratios of the actual depth values to the reference depth values comprises:
determining the second 2D coordinates by using the following functional relationship:
X2=(X1*d)/D,
Y2=(Y1*d)/D,
where, X2 is a coordinate value of a second 2D coordinate in a first direction, and X1 is a coordinate value of a first 2D coordinate in the first direction;
Y2 is a coordinate value of the second 2D coordinate in a second direction, and Y1 is a coordinate value of the first 2D coordinate in the second direction, the second direction being perpendicular to the first direction; and
d is an actual depth value, and D is a reference depth value.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
obtaining the first 2D coordinates according to second 3D features of the key points and an optical center position corresponding to the image, the second 3D feature comprising: third 2D coordinates obtained based on a 2D image and actual depth values obtained based on a depth image.

### Claim 5 (depends on 4)

The method of claim 4, wherein
obtaining the first 2D coordinates according to the second 3D features of the key points and the optical center position corresponding to the image comprises:
moving the second 3D features of the key points, such that a 3D feature of a reference point in the key points is translated to the optical center position to obtain third 3D features of the key points; and
projecting the third 3D features to a 2D imaging plane to obtain the first 2D coordinates.

### Claim 6 (depends on 5)

The method of claim 5, wherein
if the target is a body skeleton, the first 3D feature of the reference point is determined based on second 3D features of two crotch key points in the key points.

### Claim 7 (depends on 5)

The method of claim 5, wherein obtaining, based on the first 3D features, the 3D posture of the target comprises:
subtracting a depth value of the reference point from depth values corresponding to the second 2D coordinates of the key points to obtain fourth 2D coordinates and depth values corresponding to the fourth 2D coordinates;
performing normalization processing on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinates to obtain normalized fourth 2D coordinates and depth values corresponding to the normalized fourth 2D coordinates; and
processing the normalized fourth 2D coordinates and the depth values corresponding to the normalized fourth 2D coordinates by using a deep learning module to obtain the 3D posture of the target.

### Claim 8 (depends on 7)

The method of claim 7, wherein
performing the normalization processing on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinate to obtain the normalized fourth 2D coordinates and the depth values corresponding to the normalized fourth 2D coordinates comprises:
obtaining, based on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinate, coordinate means and variances of the key points; and
obtaining the normalized fourth 2D coordinates according to the coordinate mean values and the variances as well as the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinates.

### Claim 9 (depends on 4)

The method of claim 4, further comprising:
performing, based on the actual depth values, an iterative operation of projecting the 3D posture to a 2D plane to obtain fifth 2D coordinates having a minimum distance with the third 2D coordinates; and
obtaining a rotation parameter and a translation parameter of the target according to the fifth 2D coordinates and the first 3D features.

### Claim at position 10

-18. (canceled)

### Claim 19

A computer storage medium, having computer executable codes stored thereon, wherein the computer executable codes, when being executed, cause to implement a data processing method, the method comprising:
converting, according to actual depth values of key points of a target in an image and reference depth values, first two-dimensional (2D) coordinates of the key points into second 2D coordinates, the second 2D coordinates and the reference depth values forming first 3D features of the key points; and
obtaining, based on the first 3D features, a 3D posture of the target.

### Claim 20

(canceled)

### Claim 21

An electronic device, comprising:
a memory, configured to store information; and
a processor, connected to the memory and configured to implement, by executing computer executable instructions stored on the memory, a data processing method, the method comprising:
converting, according to actual depth values of key points of a target in an image and reference depth values, first two-dimensional (2D) coordinates of the key points into second 2D coordinates, wherein the second 2D coordinates and the reference depth values form first 3D features of the key points; and
obtaining, based on the first 3D features, a 3D posture of the target.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein
converting, according to the actual depth values of the key points of the target in the image and the reference depth values, the first 2D coordinates of the key points into the second 2D coordinates comprises:
obtaining the second 2D coordinates according to the first 2D coordinates and ratios of the actual depth values to the reference depth values.

### Claim 23 (depends on 22)

The electronic device of claim 22, wherein
obtaining the second 2D coordinates according to the first 2D coordinates and the ratios of the actual depth values to the reference depth values comprises:
determining the second 2D coordinates by using the following functional relationship:
X2=(X1*d)/D,
Y2=(Y1*d)/D,
where, X2 is a coordinate value of a second 2D coordinate in a first direction, and X1 is a coordinate value of a first 2D coordinate in the first direction;
Y2 is a coordinate value of the second 2D coordinate in a second direction, and Y1 is a coordinate value of the first 2D coordinate in the second direction, the second direction being perpendicular to the first direction; and
d is an actual depth value, and D is a reference depth value.

### Claim 24 (depends on 21)

The electronic device of claim 21, wherein the processor is further configured to implement:
obtaining the first 2D coordinates according to second 3D features of the key points and an optical center position corresponding to the image, the second 3D feature comprising: third 2D coordinates obtained based on a 2D image and actual depth values obtained based on a depth image.

### Claim 25 (depends on 24)

The electronic device of claim 24, wherein
obtaining the first 2D coordinates according to the second 3D features of the key points and the optical center position corresponding to the image comprises:
moving the second 3D features of the key points, such that a 3D feature of a reference point in the key points is translated to the optical center position to obtain third 3D features of the key points; and
projecting the third 3D features to a 2D imaging plane to obtain the first 2D coordinates.

### Claim 26 (depends on 25)

The electronic device of claim 25, wherein
if the target is a body skeleton, the first 3D feature of the reference point is determined based on second 3D features of two crotch key points in the key points.

### Claim 27 (depends on 25)

The electronic device of claim 25, wherein obtaining, based on the first 3D features, the 3D posture of the target comprises:
subtracting a depth value of the reference point from depth values corresponding to the second 2D coordinates of the key points to obtain fourth 2D coordinates and depth values corresponding to the fourth 2D coordinates;
performing normalization processing on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinates to obtain normalized fourth 2D coordinates and depth values corresponding to the normalized fourth 2D coordinates; and
processing the normalized fourth 2D coordinates and the depth values corresponding to the normalized fourth 2D coordinates by using a deep learning module to obtain the 3D posture of the target.

### Claim 28 (depends on 27)

The electronic device of claim 27, wherein
performing the normalization processing on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinate to obtain the normalized fourth 2D coordinates and the depth values corresponding to the normalized fourth 2D coordinates comprises:
obtaining, based on the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinate, coordinate means and variances of the key points; and
obtaining the normalized fourth 2D coordinates according to the coordinate mean values and the variances as well as the fourth 2D coordinates and the depth values corresponding to the fourth 2D coordinates.

### Claim 29 (depends on 24)

The electronic device of claim 24, wherein the processor is further configured to implement:
performing, based on the actual depth values, an iterative operation of projecting the 3D posture to a 2D plane to obtain fifth 2D coordinates having a minimum distance with the third 2D coordinates; and
obtaining a rotation parameter and a translation parameter of the target according to the fifth 2D coordinates and the first 3D features.