# US 12,118,766 B2 · Method and apparatus for detecting keypoints of human body, electronic device and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,766 B2 - Document ID: US12118766B2 - Status: Granted - Kind: B2 - Application number: 17/507,850 - Filed: 2021-10-22 - Published: 2022-02-10 - Granted: 2024-10-15 - Earliest claimed: 2019-07-15 - Assignees: Shenzhen Sensetime Technology Co., Ltd. - Applicants: Shenzhen Sensetime Technology Co., Ltd. - Inventors: Wentao Liu; Yujing Guo; Yongwang Wang; Chen Qian; Jiahua Li - Web page: https://patentgrep.com/patent/US12118766B2 ## CPC - G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]} - G06F 18/25: Fusion techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/82: using neural networks - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and an apparatus for detecting key points of a human body, and a storage medium are provided. The method includes: extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions. ## Claims ### Claim 1 A method for detecting body keypoints, comprising; extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions, the method further comprising: acquiring a first body motion state; describing, through first 3D pose data, a change of body keypoint positions corresponding to the first body motion state; and generating a first control instruction according to the first 3D pose data, and sending the first control instruction to a receiving side device, so as to display an action simulating operation corresponding to the first body motion state on a display screen of the receiving side device, or acquiring a second body motion state; describing, through second 3D pose data, a change of body keypoint positions corresponding to the second body motion state; comparing the second 3D pose data with pre-configured pose data, and generating a second control instruction in response to an inconsistent comparison result; and sending prompt information according to the second control instruction, so as to adjust the second body motion state to a target state according to the prompt information, or acquiring a third body motion state; describing, through third 3D pose data, a change of body keypoint positions corresponding to the third body motion state; and sending the third 3D pose data to the receiving side device, so as to display, on the display screen of the receiving side device, operations of a virtual avatar sampling the third 3D pose data. ### Claim 2 (depends on 1) The method of claim 1, wherein before performing the body keypoint feature fusion on the 2D pose data and the depth data corresponding to the body keypoint positions, the method further comprises: performing data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the each frame of image to obtain an RGB depth (RGBD) image data stream. ### Claim 3 (depends on 2) The method of claim 2, wherein before performing body keypoint feature fusion on the 2D pose data and the depth data that corresponds to the body keypoint positions, the method further comprises: obtaining multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed. ### Claim 4 (depends on 1) The method of claim 1, wherein detecting that the image comprises the body comprises: obtaining multiple image features by performing first image processing on a present frame of image; and in response to determining that the multiple image features are body keypoint features by using a body identification network, detecting that the present frame of image comprises the body till detection on at least one frame of image is completed. ### Claim 5 (depends on 4) The method of claim 4, wherein a training process of the body identification network comprises: taking pre-labeled body keypoint features as training sample data, inputting the training sample data to a to-be-trained body identification network for training till an output result meets a network training condition, and obtaining a trained body identification network. ### Claim 6 An apparatus for detecting body keypoints, comprising; a processor; and a memory configured to store instructions executable by the processor, wherein the processor is configured to: extract, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and perform body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions, wherein the processor is further configured to: acquire a first body motion state; describe, through first 3D pose data, a change of body keypoint positions corresponding to the first body motion state; and generate a first control instruction according to the first 3D pose data, and send the first control instruction to a receiving side device, so as to display an action simulating operation corresponding to the first body motion state on a display screen of the receiving side device, or acquire a second body motion state; describe, through second 3D pose data, a change of body keypoint positions corresponding to the second body motion state; compare the second 3D pose data with pre-configured pose data, and generate a second control instruction in response to an inconsistent comparison result; and send prompt information according to the second control instruction, so as to adjust the second body motion state to a target state according to the prompt information, or acquire a third body motion state; describe, through third 3D pose data, a change of body keypoint positions corresponding to the third body motion state; and send the third 3D pose data to the receiving side device, so as to display, on the display screen of the receiving side device, operations of a virtual avatar sampling the third 3D pose data. ### Claim 7 (depends on 6) The apparatus of claim 6, wherein the processor is further configured to: perform data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the each frame of image to obtain an RGB depth (RGBD) image data stream. ### Claim 8 (depends on 7) The apparatus of claim 7, wherein the processor is further configured to: obtain multiple image features by performing first image processing on a present frame of image; and in response to determining that the multiple image features are body keypoint features by using a body identification network, detect that the present frame of image comprises the body till detection on at least one frame of image is completed. ### Claim 9 (depends on 8) The apparatus of claim 8, wherein the processor is further configured to: take pre-labeled body keypoint features as training sample data in a training process of the body identification network, input the training sample data to a to-be-trained body identification network for training till an output result meets a network training condition, and obtain a trained body identification network. ### Claim 10 (depends on 7) The apparatus of claim 7, wherein the processor is further configured to: obtain multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed. ### Claim 11 A non-transitory computer-readable storage medium, having computer program instructions stored thereon, wherein the computer program instructions, when being executed by a processor, cause to implement operations, the operations comprising: extracting, in response to detecting that an image comprises a body, two-dimensional (2D) coordinate data for identifying body keypoint positions in the image to obtain 2D pose data; and performing body keypoint feature fusion on the 2D pose data and depth data corresponding to the body keypoint positions to obtain three-dimensional (3D) pose data for identifying the body keypoint positions, the operations further comprising: acquiring a first body motion state; describing, through first 3D pose data, a change of body keypoint positions corresponding to the first body motion state; and generating a first control instruction according to the first 3D pose data, and sending the first control instruction to a receiving side device, so as to display an action simulating operation corresponding to the first body motion state on a display screen of the receiving side device, or acquiring a second body motion state; describing, through second 3D pose data, a change of body keypoint positions corresponding to the second body motion state; comparing the second 3D pose data with pre-configured pose data, and generating a second control instruction in response to an inconsistent comparison result; and sending prompt information according to the second control instruction, so as to adjust the second body motion state to a target state according to the prompt information, or acquiring a third body motion state; describing, through third 3D pose data, a change of body keypoint positions corresponding to the third body motion state; and sending the third 3D pose data to the receiving side device, so as to display, on the display screen of the receiving side device, operations of a virtual avatar sampling the third 3D pose data. ### Claim 12 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein before performing the body keypoint feature fusion on the 2D pose data and the depth data corresponding to the body keypoint positions, the operations further comprise: performing data alignment preprocessing on each frame of image in a Red Green Blue (RGB) image data stream and depth data corresponding to the each frame of image to obtain an RGB depth (RGBD) image data stream. ### Claim 13 (depends on 12) The non-transitory computer-readable storage medium of claim 12, wherein before performing body keypoint feature fusion on the 2D pose data and the depth data that corresponds to the body keypoint positions, the operations further comprise: obtaining multiple depth data by performing second image processing on a present frame of image till image processing on at least one frame of image is completed. ### Claim 14 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the operation of detecting that the image comprises the body comprises: obtaining multiple image features by performing first image processing on a present frame of image; and in response to determining that the multiple image features are body keypoint features by using a body identification network, detecting that the present frame of image comprises the body till detection on at least one frame of image is completed. ### Claim 15 (depends on 14) The non-transitory computer-readable storage medium of claim 14, wherein a training process of the body identification network comprises: taking pre-labeled body keypoint features as training sample data, inputting the training sample data to a to-be-trained body identification network for training till an output result meets a network training condition, and obtaining a trained body identification network.