# US 2026/0073726 A1 · Hand-tracking with ir camera for xr systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073726 A1
- Document ID: US20260073726A1
- Status: Published application
- Kind: A1
- Application number: 19/396,162
- Filed: 2025-11-20
- Published: 2026-03-12
- Earliest claimed: 2022-12-08
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Piotr Gurgul; Sharon Moll; Pawel Wawruch
- Web page: https://patentgrep.com/patent/US20260073726A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 40/11: {Hand-related biometrics; Hand pose recognition}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06T 7/251: {involving models}
- G06T 7/75: {involving models}
- G06V 20/20: in augmented reality scenes
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- H04N 23/20: for generating image signals from infrared radiation only
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2207/10048: Infrared image
- G06T 2207/30196: Human being; Person
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An extended Reality (XR) system provides methodologies for capturing hand poses being made by a user in low-light environments. The XR system capture, using one or more visible light cameras, tracking video frame data of a hand pose of a user of the XR system. The XR system generates hand-tracking data based on the tracking video frame data. The hand-tracking data includes a skeletal model and a hand-tracking confidence level indicating a probability that the skeletal model matches the hand pose. The XR system compares the hand-tracking confidence level to a threshold confidence value, and based on determining the first hand-tracking confidence level is below the threshold confidence value, activates one or more wide-spectrum cameras to capture subsequent tracking video frame data of the hand pose. The XR system may also activate an IR light emitter to illuminate the hands of the user.

## Claims

### Claim 1

A method comprising:
determining a first hand-tracking confidence level of a pose of a hand of a user using first video frame data captured using one or more visible light cameras of an XR system;
in response to determining the first hand-tracking confidence level falls below a threshold confidence value, activating one or more wide-spectrum cameras of the XR system;
determining a second hand-tracking confidence level of the pose of the hand of the user using second video frame data captured using the one or more wide-spectrum camera; and
in response to determining the second hand-tracking confidence level is below the threshold confidence value, activating one or more light emitters of the XR system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first hand-tracking confidence level indicates a probability that a skeletal model generated using the first video frame data matches the pose of the hand of the user.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second hand-tracking confidence level indicates a probability that a skeletal model generated using the second video frame data matches the pose of the hand of the user.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more light emitters are Infrared (IR) light emitters and the one or more wide-spectrum cameras are sensitive to IR light.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using the pose of the hand of a user input to an XR user interface when the first hand-tracking confidence level is at or above the threshold confidence value.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using the pose of the hand of a user input to an XR user interface when the second hand-tracking confidence level is at or above the threshold confidence value.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the XR system comprises a head-wearable apparatus.

### Claim 8

A machine comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:
determining a first hand-tracking confidence level of a pose of a hand of a user using first video frame data captured using one or more visible light cameras of an XR system;
in response to determining the first hand-tracking confidence level falls below a threshold confidence value, activating one or more wide-spectrum cameras of the XR system;
determining a second hand-tracking confidence level of the pose of the hand of the user using second video frame data captured using the one or more wide-spectrum camera; and
in response to determining the second hand-tracking confidence level is below the threshold confidence value, activating one or more light emitters of the XR system.

### Claim 9 (depends on 8)

The machine of claim 8, wherein the first hand-tracking confidence level indicates a probability that a skeletal model generated using the first video frame data matches the pose of the hand of the user.

### Claim 10 (depends on 8)

The machine of claim 8, wherein the second hand-tracking confidence level indicates a probability that a skeletal model generated using the second video frame data matches the pose of the hand of the user.

### Claim 11 (depends on 8)

The machine of claim 8, wherein the one or more light emitters are Infrared (IR) light emitters and the one or more wide-spectrum cameras are sensitive to IR light.

### Claim 12 (depends on 8)

The machine of claim 8, wherein the operations further comprise:
using the first hand-tracking confidencing pose of the hand of a user input to an XR user interface when the first hand-tracking confidence level is at or above the threshold confidence value.

### Claim 13 (depends on 8)

The machine of claim 8, wherein the operations further comprise:
using the second hand-tracking confidencing pose of the hand of a user input to an XR user interface when the second hand-tracking confidence level is at or above the threshold confidence value.

### Claim 14 (depends on 8)

The machine of claim 8, wherein the XR system comprises a head-wearable apparatus.

### Claim 15

A machine-storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
determining a first hand-tracking confidence level of a pose of a hand of a user using first video frame data captured using one or more visible light cameras of an XR system;
in response to determining the first hand-tracking confidence level falls below a threshold confidence value, activating one or more wide-spectrum cameras of the XR system;
determining a second hand-tracking confidence level of the pose of the hand of the user using second video frame data captured using the one or more wide-spectrum camera; and
in response to determining the second hand-tracking confidence level is below the threshold confidence value, activating one or more light emitters of the XR system.

### Claim 16 (depends on 15)

The machine-storage medium of claim 15, wherein the first hand-tracking confidence level indicates a probability that a skeletal model generated using the first video frame data matches the pose of the hand of the user.

### Claim 17 (depends on 15)

The machine-storage medium of claim 15, wherein the second hand-tracking confidence level indicates a probability that a skeletal model generated using the second video frame data matches the pose of the hand of the user.

### Claim 18 (depends on 15)

The machine-storage medium of claim 15, wherein the one or more light emitters are Infrared (IR) light emitters and the one or more wide-spectrum cameras are sensitive to IR light.

### Claim 19 (depends on 15)

The machine-storage medium of claim 15, wherein the operations further comprise:
using the first hand-tracking confidencing pose of the hand of a user input to an XR user interface when the first hand-tracking confidence level is at or above the threshold confidence value.

### Claim 20 (depends on 15)

The machine-storage medium of claim 15, wherein the operations further comprise:
using the second hand-tracking confidencing pose of the hand of a user input to an XR user interface when the second hand-tracking confidence level is at or above the threshold confidence value.
