Hand pose-dependent scale estimation for extended reality

An extended reality (XR) device comprising:

• at least one processor;
• and
• at least one memory storing instructions that, when executed by the at least one processor, cause the XR device to perform operations comprising:
• capturing, via one or more optical sensors of the XR device, at least one image of a hand of a user of the XR device;
• processing the at least one image to detect a hand pose relative to the XR device;
• accessing one or more existing hand scale estimates, each uniquely associated with a respective hand pose;
• determining that the detected hand pose does not correspond to any of the one or more existing hand scale estimates;
• based on determining that the detected hand pose does not correspond to any of the one or more existing hand scale estimates:
• triggering a calibration operation to obtain a hand scale estimate associated with the detected hand pose, and
• updating the one or more existing hand scale estimates to add the hand scale estimate associated with the detected hand pose;
• applying the hand scale estimate associated with the detected hand pose to generate positional data for one or more features of the hand of the user;
• and
• tracking, based on the positional data, the hand of the user while the user is using the XR device.

One or more non-transitory computer-readable storage media, the one or more non-transitory computer-readable storage media including instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• capturing, via one or more optical sensors of an extended reality (XR) device, at least one image of a hand of a user of the XR device;
• processing the at least one image to detect a hand pose relative to the XR device;
• accessing one or more existing hand scale estimates, each uniquely associated with a respective hand pose;
• determining that the detected hand pose does not correspond to any of the one or more existing hand scale estimates;
• based on determining that the detected hand pose does not correspond to any of the one or more existing hand scale estimates:
• triggering a calibration operation to obtain a hand scale estimate associated with the detected hand pose, and
• updating the one or more existing hand scale estimates to add the hand scale estimate associated with the detected hand pose;
• applying the hand scale estimate associated with the detected hand pose to generate positional data for one or more features of the hand of the user;
• and
• tracking, based on the positional data, the hand of the user while the user is using the XR device.