# US 2024/0203152 A1 · Method for identifying human poses in an image, computer system, and non-transitory computer-readable medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203152 A1
- Document ID: US20240203152A1
- Status: Published application
- Kind: A1
- Application number: 18/559,082
- Filed: 2020-12-14
- Published: 2024-06-20
- Earliest claimed: 2020-12-14
- Assignees: Innopeak Technology, Inc.
- Applicants: Innopeak Technology, Inc.
- Inventors: Yuliang GUO; Zhong LI; Xiangyu DU; Yi XU; Shuxue QUAN
- Web page: https://patentgrep.com/patent/US20240203152A1

## CPC

- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06T 7/11: Region-based segmentation
- G06T 7/50: Depth or shape recovery
- G06T 7/73: using feature-based methods
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for identifying human poses in an image, a computer system, and a non-transitory computer-readable medium are provided. The method includes obtaining the image, encoding the image into a feature map, converting the feature map into a plurality of TPDF maps corresponding to a plurality of predefined part types, and generating a global pose map based on the feature map and the plurality of TPDF maps corresponding to the predefined part types. Each TPDF map corresponds to a respective predefined part type and represents a plurality of vectors connecting each position on the image to a closest body part among the one or more body parts of the respective predefined part type. A global pose map is generated based on the feature map and the plurality of TPDF maps, identifying one or more human bodies having a plurality of body parts in a scene of the image.

## Claims

### Claim 1

A method for identifying human poses in an image, comprising:
obtaining the image;
encoding the image into a feature map;
converting the feature map into a plurality of Truncated Part Displacement Field (TPDF) maps corresponding to a plurality of predefined part types, where:
each TPDF map corresponds to a respective predefined part type, and the image comprises one or more body parts of the respective predefined part type;
each TPDF map comprises an x-axis displacement map and a y-axis displacement map, and represents a plurality of vectors connecting each position on the image to a closest body part among the one or more body parts of the respective predefined part type; and
each of the plurality of vectors is marked active when a magnitude of a respective vector is below a predefined vector truncating threshold; and
generating a global pose map based on the feature map and the plurality of TPDF maps corresponding to the predefined part types, wherein the global pose map is configured to identify one or more human bodies having a plurality of body parts in a scene of the image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the global pose map comprises a position of each body part, the method further comprising:
converting the feature map into a plurality of confidence maps corresponding to the plurality of body parts in the image, each confidence map comprising a confidence level of a respective body part in each position on the image;
for each of the plurality of body parts, generating an average positon position, comprising:
aggregating the x-axis and y-axis displacement maps of the TPDF map based on the confidence map, respectively;
based on the aggregation of the TPDF map, adjusting the position of each body part in the global pose map to generate the average position of the respective body part.

### Claim 3 (depends on 2)

The method of claim 2, further comprising for one of the plurality of body parts:
determining that a corresponding confidence map corresponds to a confidence level less than a confidence level threshold; and
aborting adjusting the position of the one of the plurality of body parts, such that the position of the one of the plurality of body parts is approximately equal to the average position of the respective part type.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
converting the feature map into a plurality of depth maps corresponding to the plurality of body parts in the image, each depth map comprising depth information of a respective body part in each position on the image;
for each of the plurality of body parts, generating an average depth by aggregating the depth information of the respective depth map based on the confidence map.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of TPDF maps comprises a plurality of first TPDF maps, and generating the global pose map further comprises:
combining the feature map with the plurality of first TPDF maps to generate a combined feature map;
converting the combined feature map to a plurality of second TPDF maps; and
generating the global pose map from the combined feature map and the plurality of second TPDF maps.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
each second TPDF map corresponds to a respective predefined part type;
each second TPDF map comprises a second x-axis displacement map and a second y-axis displacement map, and represents a plurality of second vectors connecting each position on the image to a closest body part among the one or more body parts of the respective predefined part type; and
each of the plurality of second vectors is marked active when a magnitude of a respective second vector is below the predefined vector truncating threshold.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
converting the feature map into a plurality of confidence maps, the plurality of confidence maps comprising a background confidence map corresponding to a background of the image and a first set of part confidence maps corresponding to the plurality of body parts;
wherein the global pose map is generated based on a combination of the feature map, the plurality of TPDF maps, and the plurality of confidence maps.

### Claim 8 (depends on 7)

The method of claim 7, wherein the plurality of TPDF maps comprises a plurality of first TPDF maps, and the plurality of confidence maps comprises a plurality of first confidence maps, and wherein generating the global pose map further comprises:
combining the feature map, the plurality of first TPDF maps, and the plurality of first confidence maps to generate a combined feature map;
converting the combined feature map to a plurality of second TPDF maps;
converting the combined feature map to a plurality of second confidence maps; and
generating the global pose map from the combined feature map, the plurality of second TPDF maps, and the plurality of second confidence maps.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
converting the feature map to into a plurality of depth maps, each depth map corresponding to a respective body part; and
wherein the global pose map is generated based on a combination of the feature map, the plurality of TPDF maps, the plurality of confidence maps, and the plurality of depth maps.

### Claim 10 (depends on 9)

The method of claim 9, wherein:
the plurality of TPDF maps comprises a plurality of first TPDF maps;
the plurality of confidence maps comprises a plurality of first confidence maps;
the plurality of depth maps comprises a plurality of first depth maps; and
generating the global pose map further comprises;
combining the feature map, the plurality of first TPDF maps, the plurality of first confidence maps, and the plurality of first depth maps to generate a combined feature map;
converting the combined feature map to a plurality of second TPDF maps;
converting the combined feature map to a plurality of second confidence maps;
converting the combined feature map to a plurality of second depth maps; and
generating the global pose map from the combined feature map, the plurality of second TPDF maps, the plurality of second confidence maps, and the plurality of second depth maps.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the plurality of body parts comprises a first number (K) of body parts;
the plurality of TPDF maps comprises a second number (2K) of TPDF maps;
the plurality of confidence maps comprises a third number (K+1) of confidence maps; and
the plurality of depth maps comprises the first number (K) of depth maps.

### Claim 12 (depends on 10)

The method of claim 10, wherein:
the image is encoded to the feature map using a backbone network;
the confidence, TPDF, and depth maps are generated using a heatmap model, a TPDF model, and a depth model, respectively;
the global pose map is generated using a global pose network; and
the backbone network, heatmap model, TPDF model, depth model, and global pose network are trained end-to-end based on a total loss combining a heatmap loss, a TPDF loss, a depth loss, and a global pose loss.

### Claim 13 (depends on 1)

The method of claim 1, wherein the image comprises at least one of an RGB image and a depth image.

### Claim 14 (depends on 1)

The method of claim 1, wherein the image comprises two body parts of the same part type, and the two body parts belong to two distinct human bodies and partially overlap with each other.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
generating a plurality of part visibility maps corresponding to the plurality of body parts comprising the two body parts of the same part type, wherein the global pose map is generated based on the plurality of part visibility maps, thereby indicating occlusion between the two body parts of the same part type.

### Claim 16 (depends on 1)

The method of claim 1, wherein the global pose map is generated using a global pose network, and the global pose map is generated using a global pose network, and the global pose network comprises a plurality of 3×3 convolutional layers and a max-pooling layer.

### Claim 17

A computer system, comprising:
one or more processors; and
memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform a method for identifying human poses in an image;
wherein the method comprises;
obtaining the image;
encoding the image into a feature map;
converting the feature map into a plurality of Truncated Part Displacement Field (TPDF) maps corresponding to a plurality of predefined part types, where:
each TPDF map corresponds to a respective predefined part type, and the image comprises one or more body parts of the respective predefined part type;
each TPDF map comprises an x-axis displacement map and a y-axis displacement map, and represents a plurality of vectors connecting each position on the image to a closest body part among the one or more body parts of the respective predefined part type; and
each of the plurality of vectors is marked active when a magnitude of a respective vector is below a predefined vector truncating threshold; and
generating a global pose map based on the feature map and the plurality of TPDF maps corresponding to the predefined part types, wherein the global pose map is configured to identify one or more human bodies having a plurality of body parts in a scene of the image.

### Claim 18

A non-transitory computer-readable medium, having instructions stored thereon, which when executed by one or more processors cause the processors to perform a method for identifying human poses in an image;
wherein the method comprises:
obtaining the image;
encoding the image into a feature map;
converting the feature map into a plurality of Truncated Part Displacement Field (TPDF) maps corresponding to a plurality of predefined part types, where:
each TPDF map corresponds to a respective predefined part type, and the image comprises one or more body parts of the respective predefined part type;
each TPDF map comprises an x-axis displacement map and a y-axis displacement map, and represents a plurality of vectors connecting each position on the image to a closest body part among the one or more body parts of the respective predefined part type; and
each of the plurality of vectors is marked active when a magnitude of a respective vector is below a predefined vector truncating threshold; and
generating a global pose map based on the feature map and the plurality of TPDF maps corresponding to the predefined part types, wherein the global pose map is configured to identify one or more human bodies having a plurality of body parts in a scene of the image.

### Claim 19 (depends on 17)

The computer system of claim 17, wherein the global pose map comprises a position of each body part, the method further comprising:
converting the feature map into a plurality of confidence maps corresponding to the plurality of body parts in the image, each confidence map comprising a confidence level of a respective body part in each position on the image;
for each of the plurality of body parts, generating an average position, comprising:
aggregating the x-axis and y-axis displacement maps of the TPDF map based on the confidence map, respectively;
based on the aggregation of the TPDF map, adjusting the position of each body part in the global pose map to generate the average position of the respective body part.

### Claim 20 (depends on 17)

The computer system of claim 17, wherein the plurality of TPDF maps comprises a plurality of first TPDF maps, and generating the global pose map further comprises:
combining the feature map with the plurality of first TPDF maps to generate a combined feature map;
converting the combined feature map to a plurality of second TPDF maps; and
generating the global pose map from the combined feature map and the plurality of second TPDF maps.
