# US 11,908,244 B2 · Human posture detection utilizing posture reference maps

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,244 B2
- Document ID: US11908244B2
- Status: Granted
- Kind: B2
- Application number: 17/297,882
- Filed: 2019-11-20
- Published: 2022-01-06
- Granted: 2024-02-20
- Earliest claimed: 2018-11-27
- Assignees: Bigo Technology Pte. Ltd.
- Applicants: Bigo Technology Pte. Ltd.
- Inventors: Wei Xiang; Yifeng Wang; Qiushi Huang; Zhujin Liang
- Web page: https://patentgrep.com/patent/US11908244B2

## CPC

- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/251: {involving models}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Provided is a human posture detection method including: acquiring a plurality of frames of image data; acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points; identifying a human-posture key point in each of the human posture reference maps; and generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data.

## Claims

### Claim 1

A human posture detection method, comprising:
acquiring a plurality of frames of image data;
acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points;
identifying a human-posture key point in each of the human posture reference maps; and
generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data,
wherein the human posture detection model comprises a main path, a first branch, and a second branch, the main path comprising a residual module and an up-sampling module, the first branch comprising a refinement network module, and the second branch comprising a feedback module; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data into the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring a first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data;
acquiring a second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein, and acquiring a third convolution result by inputting the first convolution result output by the residual module into the refinement network module and processing the first convolution result therein; and
acquiring the plurality of human posture reference maps by adding up the second convolution result and the third convolution result.

### Claim 2 (depends on 1)

The method according to claim 1, wherein acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data to the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting, in response to the human posture confidence maps of the previous frame of image data being credible, the current frame of image data and the human posture confidence maps of the previous frame of image data into the human posture detection model; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive by inputting, in response to the human posture confidence maps of the previous frame of image data being incredible, the current frame of image data and target image data into the human posture detection model, wherein the target image data is image data containing no prior knowledge.

### Claim 3 (depends on 1)

The method according to claim 1, wherein each of the human posture reference maps comprises a plurality of candidate points for the human-posture key point, and a coordinate position of each of the candidate points corresponds to one probability value; and
identifying the human-posture key points in each of the human posture reference maps comprises:
determining a coordinate position corresponding to a maximum probability value among a plurality of probability values, and taking a candidate point corresponding to the coordinate position as the human-posture key point.

### Claim 4 (depends on 2)

The method according to claim 2, wherein generating the human posture confidence maps of the current frame of image data based on the credibility of the human-posture key points comprises:
generating, in response to the human-posture key points being credible, mask patterns with the human-posture key points as centers, and taking the mask patterns as the human posture confidence maps; and
taking, in response to the human-posture key points being incredible, the target image data as the human posture confidence maps.

### Claim 5 (depends on 4)

The method according to claim 4, further comprising:
determining, in response to probability values corresponding to the human-posture key points being greater than a preset threshold value, that the human-posture key points are credible; and
determining, in response to the probability values corresponding to the human-posture key points being less than or equal to the preset threshold value, that the human-posture key points are incredible.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the residual module comprises a first residual unit, a second residual unit, and a third residual unit; and
acquiring the first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data comprises:
acquiring a first intermediate result by inputting the current frame image data into the first residual unit and processing the current frame image data therein;
inputting the human posture confidence maps of the previous image data into the feedback module and processing the human posture confidence maps of the previous image data therein;
acquiring an addition result by adding up the first intermediate result and a processing result output by the feedback module;
acquiring a second intermediate result by inputting the addition result into the second residual unit and processing the addition result therein; and
acquiring a third intermediate result by inputting the second intermediate result into the third residual unit and processing the second intermediate result therein, and taking the third intermediate result as the first convolution result;
wherein numbers of channels for the first intermediate result, the second intermediate result, and the third intermediate result are increased successively.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the human posture detection model further comprises a third branch; and
acquiring the second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein comprise:
acquiring a fourth intermediate result by inputting the first intermediate result into the third branch and processing the first intermediate result therein;
acquiring a fifth intermediate result by inputting the second intermediate result into the third branch and processing the second intermediate result therein;
acquiring a sixth intermediate result by inputting the third intermediate result and the fifth intermediate result into the up-sampling module and processing the third intermediate result and the fifth intermediate result therein; and
acquiring a seventh intermediate result by inputting the fourth intermediate result and the sixth intermediate result into the up-sampling module and processing the fourth intermediate result and the sixth intermediate result therein, and taking the seventh intermediate result as the second convolution result;
wherein numbers of channels for the sixth intermediate result and the seventh intermediate result are decreased successively.

### Claim 8 (depends on 1)

The method according to claim 1, acquiring the plurality of human posture reference maps by adding up the second convolution result and the third convolution result further comprises:
acquiring a target result by adding up the first convolution result and the second convolution result; and
acquiring the plurality of human posture reference maps by adding up the plurality of human posture reference maps and the target result.

### Claim 9 (depends on 1)

The method according to claim 1, wherein the human posture detection model is generated by training using a convolutional neutral network applicable to an embedded platform.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising:
inputting, in response to the current frame of image data being not the last frame of image data, the human posture confidence maps of the current frame of image data into the human posture detection model; and
terminating, in response to the current frame of image data being the last frame of image data, generation of human posture confidence maps of the plurality of frames of image data.

### Claim 11

A device, comprising:
at least one processor; and
a memory configured to store at least one program,
wherein the at least one processor, when running the at least one program, is caused to perform a human posture detection method comprising:
acquiring a plurality of frames of image data;
acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points;
identifying a human-posture key point in each of the human posture reference maps; and
generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data,
wherein the human posture detection model comprises a main path, a first branch, and a second branch, the main path comprising a residual module and an up-sampling module, the first branch comprising a refinement network module, and the second branch comprising a feedback module; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data into the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring a first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data;
acquiring a second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein, and acquiring a third convolution result by inputting the first convolution result output by the residual module into the refinement network module and processing the first convolution result therein; and
acquiring the plurality of human posture reference maps by adding up the second convolution result and the third convolution result.

### Claim 12 (depends on 11)

The device according to claim 11, wherein acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data to the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting, in response to the human posture confidence maps of the previous frame of image data being credible, the current frame of image data and the human posture confidence maps of the previous frame of image data into the human posture detection model; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive by inputting, in response to the human posture confidence maps of the previous frame of image data being incredible, the current frame of image data and target image data into the human posture detection model, wherein the target image data is image data containing no prior knowledge.

### Claim 13 (depends on 12)

The device according to claim 12, wherein generating the human posture confidence maps of the current frame of image data based on the credibility of the human-posture key points comprises:
generating, in response to the human-posture key points being credible, mask patterns with the human-posture key points as centers, and taking the mask patterns as the human posture confidence maps; and
taking, in response to the human-posture key points being incredible, the target image data as the human posture confidence maps.

### Claim 14 (depends on 11)

The device according to claim 11, wherein each of the human posture reference maps comprises a plurality of candidate points for the human-posture key point, and a coordinate position of each of the candidate points corresponds to one probability value; and
identifying the human-posture key points in each of the human posture reference maps comprises:
determining a coordinate position corresponding to a maximum probability value among a plurality of probability values, and taking a candidate point corresponding to the coordinate position as the human-posture key point.

### Claim 15 (depends on 11)

The device according to claim 11, wherein the residual module comprises a first residual unit, a second residual unit, and a third residual unit; and
acquiring the first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data comprises:
acquiring a first intermediate result by inputting the current frame image data into the first residual unit and processing the current frame image data therein;
inputting the human posture confidence maps of the previous image data into the feedback module and processing the human posture confidence maps of the previous image data therein;
acquiring an addition result by adding up the first intermediate result and a processing result output by the feedback module;
acquiring a second intermediate result by inputting the addition result into the second residual unit and processing the addition result therein; and
acquiring a third intermediate result by inputting the second intermediate result into the third residual unit and processing the second intermediate result therein, and taking the third intermediate result as the first convolution result;
wherein numbers of channels for the first intermediate result, the second intermediate result, and the third intermediate result are increased successively.

### Claim 16 (depends on 15)

The device according to claim 15, wherein the human posture detection model further comprises a third branch; and
acquiring the second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein comprise:
acquiring a fourth intermediate result by inputting the first intermediate result into the third branch and processing the first intermediate result therein;
acquiring a fifth intermediate result by inputting the second intermediate result into the third branch and processing the second intermediate result therein;
acquiring a sixth intermediate result by inputting the third intermediate result and the fifth intermediate result into the up-sampling module and processing the third intermediate result and the fifth intermediate result therein; and
acquiring a seventh intermediate result by inputting the fourth intermediate result and the sixth intermediate result into the up-sampling module and processing the fourth intermediate result and the sixth intermediate result therein, and taking the seventh intermediate result as the second convolution result;
wherein numbers of channels for the sixth intermediate result and the seventh intermediate result are decreased successively.

### Claim 17 (depends on 11)

The device according to claim 11, wherein the method further comprises:
inputting, in response to the current frame of image data being not the last frame of image data, the human posture confidence maps of the current frame of image data into the human posture detection model; and
terminating, in response to the current frame of image data being the last frame of image data, generation of human posture confidence maps of the plurality of frames of image data.

### Claim 18

A non-transitory computer-readable storage medium storing at least one computer program therein, wherein the computer program, when run by a processor, causes the processor to perform a human posture detection method comprising:
acquiring a plurality of frames of image data;
acquiring a plurality of human posture reference maps output by a human posture detection model responsive to inputting a current frame of image data to the human posture detection model with reference to human posture confidence maps of a previous frame of image data, wherein different human posture reference maps correspond different human-posture key points;
identifying a human-posture key point in each of the human posture reference maps; and
generating human posture confidence maps of the current frame of image data based on credibility of the human-posture key points, wherein the human posture confidence maps of the current frame of image data is configured to participate in generation of human posture confidence maps of a next frame of image data,
wherein the human posture detection model comprises a main path, a first branch, and a second branch, the main path comprising a residual module and an up-sampling module, the first branch comprising a refinement network module, and the second branch comprising a feedback module; and
acquiring the plurality of human posture reference maps output by the human posture detection model responsive to inputting the current frame of image data into the human posture detection model with reference to the human posture confidence maps of the previous frame of image data comprises:
acquiring a first convolution result output by the residual module based on a processing result of the residual module on the current frame of image data and a processing result of the feedback module on the human posture confidence maps of the previous image data;
acquiring a second convolution result by inputting the first convolution result output by the residual module into the up-sampling module and processing the first convolution result therein, and acquiring a third convolution result by inputting the first convolution result output by the residual module into the refinement network module and processing the first convolution result therein; and
acquiring the plurality of human posture reference maps by adding up the second convolution result and the third convolution result.