Scalable Real-Time Hand Tracking

A computing system for hand tracking, the system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
• obtaining a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;
• processing the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;
• and
• processing the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes.

A computer-implemented method for hand tracking, the method comprising:

• obtaining, by a computing system comprising one or more processors, a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;
• processing, by the computing system, the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;
• and
• processing, by the computing system, the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• obtaining a first image frame, wherein the first image frame is descriptive of a hand comprising a palm;
• processing the first image frame with a machine-learned palm detection model to generate one or more oriented bounding boxes associated with a position of the palm, wherein the position of the palm is determined with the machine-learned palm detection model based on one or more features in the first image frame, wherein the one or more oriented bounding boxes are descriptive of an orientation of at least one of the palm or the hand;
• and
• processing the one or more oriented bounding boxes with a machine-learned hand landmark model to determine a first plurality of hand landmark positions within the first image frame based at least in part on the one or more oriented bounding boxes.