Methods and systems for 3d hand pose estimation from rgb images

A computer-implemented method for estimating a set of 3D keypoints representing a 3D hand pose comprising:

• processing a 2D image containing a detected hand using a U-net network to obtain a global feature vector and a heatmap for each of the keypoints;
• concatenating information from the global feature vector and the heatmap to obtain a set of input tokens;
• processing the input tokens using a transformer encoder to obtain a first set of 2D keypoints representing estimated 2D locations of the keypoints in a first 2D view;
• inputting the first set of 2D keypoints as a query to a transformer decoder, with cross-attention from the transformer encoder, to obtain a second set of 2D keypoints representing estimated 2D locations of the keypoints in a second 2D view;
• and
• aggregating the first and second sets of 2D keypoints to output the set of estimated 3D keypoints.

A non-transitory computer-readable medium having instructions encoded thereon, wherein the instructions are executable by a processing unit of a computing system to cause the computing system to perform operations of:

• processing a 2D image containing a detected hand using a U-net network to obtain a global feature vector and a heatmap for each of the keypoints;
• concatenating information from the global feature vector and the heatmap to obtain a set of input tokens;
• processing the input tokens using a transformer encoder to obtain a first set of 2D keypoints representing estimated 2D locations of the keypoints in a first 2D view;
• inputting the first set of 2D keypoints as a query to a transformer decoder, with cross-attention from the transformer encoder, to obtain a second set of 2D keypoints representing estimated 2D locations of the keypoints in a second 2D view;
• and
• aggregating the first and second sets of 2D keypoints to output the set of estimated 3D keypoints.