Depth-Based 3D Human Pose Detection and Tracking

A system comprising:

• a processor;
• and
• a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
• determining, for each respective keypoint of a plurality of keypoints, a corresponding initial three-dimensional (3D) position of the respective keypoint, wherein the plurality of keypoints represents a corresponding plurality of predetermined body locations of a body of an actor;
• receiving sensor data representing the body of the actor and comprising (i) image data and (ii) depth data;
• determining, for each respective keypoint of the plurality of keypoints, a corresponding visibility value based on a visibility of the respective keypoint in the image data;
• determining, for each respective keypoint of the plurality of keypoints, a corresponding depth field value based on a distance between the corresponding initial 3D position and a corresponding reference 3D position that is based on at least one nearest neighbor of the corresponding initial 3D position in the depth data;
• determining, based on the corresponding visibility value and the corresponding depth field value of each respective keypoint of the plurality of keypoints, a loss value;
• and
• determining, for each respective keypoint of the plurality of keypoints, a corresponding updated 3D position of the respective keypoint based on the loss value.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

• determining, for each respective keypoint of a plurality of keypoints, a corresponding initial three-dimensional (3D) position of the respective keypoint, wherein the plurality of keypoints represents a corresponding plurality of predetermined body locations of a body of an actor;
• receiving sensor data representing the body of the actor and comprising (i) image data and (ii) depth data;
• determining, for each respective keypoint of the plurality of keypoints, a corresponding visibility value based on a visibility of the respective keypoint in the image data;
• determining, for each respective keypoint of the plurality of keypoints, a corresponding depth field value based on a distance between the corresponding initial 3D position and a corresponding reference 3D position that is based on at least one nearest neighbor of the corresponding initial 3D position in the depth data;
• determining, based on the corresponding visibility value and the corresponding depth field value of each respective keypoint of the plurality of keypoints, a loss value;
• and
• determining, for each respective keypoint of the plurality of keypoints, a corresponding updated 3D position of the respective keypoint based on the loss value.