# US 2026/0094472 A1 · Monocular Hand Tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094472 A1
- Document ID: US20260094472A1
- Status: Published application
- Kind: A1
- Application number: 19/330,038
- Filed: 2025-09-16
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Pablo Caballero Garces; Eslam A. Mostafa; Timothy M. Clifford
- Web page: https://patentgrep.com/patent/US20260094472A1

## CPC

- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 40/11: {Hand-related biometrics; Hand pose recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various implementations use one or more images from a single camera and enrolled hand data to predict a three-dimensional (3D) position and configuration of a hand. The enrollment data may provide information about 3D hand shape and size (e.g., fixed/actual distances between joints). Such information may facilitate determining depth and other 3D characteristics for the 3D position and configuration using images from a single camera (i.e., using a 2D image from a single camera and without requiring triangulation using live images from multiple cameras or multiple viewpoints). The hand's 3D position and configuration may be represented in a way (e.g., a format) that is appropriate for Bayesian optimization. Some implementations may use (e.g., fit) a predetermined hand shape/size with the captured image data to predict the hand's 3D position and configuration.

## Claims

### Claim 1

A method comprising:
at an electronic device having a processor:
obtaining live data comprising an image from a camera of the electronic device, the image depicting a hand of a user at a point in time;
obtaining enrollment data corresponding to a size of at least a portion of a hand, wherein the enrollment data is based on sensor data of the hand of the user obtained prior to the live data being obtained;
determining parameters for a hand representation based on the live data and the enrollment data, wherein the parameters comprise rotation parameter values corresponding to hand joints and hand pose parameter values corresponding to hand pose; and
determining a three-dimensional (3D) position and configuration of the hand based on the parameters of the hand representation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the enrollment data comprises fixed distances between the hand joints of the hand representation, the fixed distances determined based on the sensor data regarding the hand portion of the user obtained prior to the live data being obtained.

### Claim 3 (depends on 1)

The method of claim 1, wherein the enrollment data was determined based on an enrollment in which:
multiple cameras simultaneously capture the sensor data; or
one or more cameras and a depth sensor simultaneously capture the sensor data.

### Claim 4 (depends on 1)

The method of claim 1, wherein the hand representation has a format that enables Bayesian optimization using the live data and the enrollment data.

### Claim 5 (depends on 1)

The method of claim 1, wherein the rotation parameter values correspond to angles of hand joints.

### Claim 6 (depends on 1)

The method of claim 1, wherein the hand pose parameter values correspond to hand 3D position and orientation.

### Claim 7 (depends on 6)

The method of claim 6, wherein the hand pose parameter values correspond to wrist position and wrist rotation.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the parameters for the hand representation comprises fitting the live data and enrollment data.

### Claim 9 (depends on 8)

The method of claim 8, wherein a 3D position of at least a portion of the hand is determined based on the fitting.

### Claim 10 (depends on 1)

The method of claim 1 further comprising determining 3D positions and configurations of the hand over time based on a sequence of image in the live data and the enrollment data.

### Claim 11 (depends on 10)

The method of claim 10 further comprising using a filter to smooth the 3D positions and configurations determined for the hand over time.

### Claim 12 (depends on 10)

The method of claim 10 further comprising adjusting the determined 3D positions and configurations based on determining whether the 3D positions and configurations determined for the hand over time deviate from predictions based on motion tracking.

### Claim 13 (depends on 1)

The method of claim 1, wherein the live data is obtained via an outward facing sensor on a head-mounted device (HMD) or augmented reality (AR) glasses.

### Claim 14

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising:
obtaining live data comprising an image from a camera of the electronic device, the image depicting a hand of a user at a point in time:
obtaining enrollment data corresponding to a size of at least a portion of a hand, wherein the enrollment data is based on sensor data of the hand of the user obtained prior to the live data being obtained;
determining parameters for a hand representation based on the live data and the enrollment data, wherein the parameters comprise rotation parameter values corresponding to hand joints and hand pose parameter values corresponding to hand pose; and
determining a three-dimensional (3D) position and configuration of the hand based on the parameters of the hand representation.

### Claim 15 (depends on 14)

The system of claim 14, wherein the enrollment data comprises fixed distances between the hand joints of the hand representation, the fixed distances determined based on the sensor data regarding the hand portion of the user obtained prior to the live data being obtained.

### Claim 16 (depends on 14)

The system of claim 14, wherein the enrollment data was determined based on an enrollment in which:
multiple cameras simultaneously capture the sensor data; or
one or more cameras and a depth sensor simultaneously capture the sensor data.

### Claim 17 (depends on 14)

The system of claim 14, wherein the hand representation has a format that enables Bayesian optimization using the live data and the enrollment data.

### Claim 18 (depends on 14)

The system of claim 14, wherein the rotation parameter values correspond to angles of hand joints.

### Claim 19 (depends on 14)

The system of claim 14, wherein the hand pose parameter values correspond to hand 3D position and orientation.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
obtaining live data comprising an image from a camera of the electronic device, the image depicting a hand of a user at a point in time;
obtaining enrollment data corresponding to a size of at least a portion of a hand, wherein the enrollment data is based on sensor data of the hand of the user obtained prior to the live data being obtained;
determining parameters for a hand representation based on the live data and the enrollment data, wherein the parameters comprise rotation parameter values corresponding to hand joints and hand pose parameter values corresponding to hand pose; and
determining a three-dimensional (3D) position and configuration of the hand based on the parameters of the hand representation.
