# US 2026/0064186 A1 · Dynamically orientated labels for xr user interfaces

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0064186 A1
- Document ID: US20260064186A1
- Status: Published application
- Kind: A1
- Application number: 18/823,412
- Filed: 2024-09-03
- Published: 2026-03-05
- Earliest claimed: 2024-09-03
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Viktoria Hwang; James Powderly; Karen Stolzenberg; Mathieu Emmanuel Vignau
- Web page: https://patentgrep.com/patent/US20260064186A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06V 40/107: {Static hand or arm}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An extended Reality (XR) system is provided that enhances user interaction within XR environments. The XR system captures tracking data using one or more sensors, including hand tracking data of a user's hand and pose data of the XR system itself. By continuously capturing this data, the XR system dynamically generates a hand-located user interface that includes interactive virtual objects associated with specific locations on the surface of the user's hand. Additionally, the XR system generates labels for the interactive virtual objects that are dynamically oriented toward the user as the user moves their hands and head when interacting with an XR environment.

## Claims

### Claim 1

A machine-implemented method, comprising:
capturing, using one or more sensors of an eXtended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;
while continuously capturing the tracking data and the pose data, performing operations comprising:
generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;
generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;
continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of hand movement or head movement of the user; and
providing the hand-located user interface to the user.

### Claim 2 (depends on 1)

The machine-implemented method of claim 1, wherein the surface is a dorsal surface of the hand.

### Claim 3 (depends on 1)

The machine-implemented method of claim 1, wherein the surface is a palmar surface of the hand.

### Claim 4 (depends on 1)

The machine-implemented method of claim 1, further comprising:
measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and
adjusting a size of the interactive virtual object using the distance.

### Claim 5 (depends on 4)

The machine-implemented method of claim 4, wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

### Claim 6 (depends on 4)

The machine-implemented method of claim 4, wherein the size is adjusted in steps using a fixed interval.

### Claim 7 (depends on 1)

The machine-implemented method of claim 1, wherein the XR system is a head-wearable apparatus.

### Claim 8

A machine comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the machine to perform operations comprising:
capturing, using one or more sensors of an eXtended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;
while continuously capturing the tracking data and the pose data, performing operations comprising:
generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;
generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;
continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of band movement of head movement of the user; and
providing the hand-located user interface to the user.

### Claim 9 (depends on 8)

The machine of claim 8, wherein the surface is a dorsal surface of the hand.

### Claim 10 (depends on 8)

The machine of claim 8, wherein the surface is a palmar surface of the hand.

### Claim 11 (depends on 8)

The machine of claim 8, wherein the operations further comprise:
measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and
adjusting a size of the interactive virtual object using the distance.

### Claim 12 (depends on 11)

The machine of claim 11, wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

### Claim 13 (depends on 11)

The machine of claim 11, wherein the size is adjusted in steps using a fixed interval.

### Claim 14 (depends on 8)

The machine of claim 8, wherein the XR system is a head-wearable apparatus.

### Claim 15

A machine-storage medium, the machine-storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
capturing, using one or more sensors of an eXtended Reality (XR) system, tracking data of a user, the tracking data including hand tracking data of a hand of the user and pose data of the XR system;
while continuously capturing the tracking data and the pose data, performing operations comprising:
generating, using the tracking data, a hand-located user interface including an interactive virtual object associated with a location on a surface of the hand;
generating, using the interactive virtual object and the pose data, a label associated with the interactive virtual object, the label orientated to a viewpoint of the user;
continuously adjusting an orientation of the label based on changes of the viewpoint of the user and a position of the hand to maintain readability of the label regardless of hand movement or head movement of the user; and
providing the hand-located user interface to the user.

### Claim 16 (depends on 15)

The machine-storage medium of claim 15, wherein the surface is a dorsal surface of the hand.

### Claim 17 (depends on 15)

The machine-storage medium of claim 15, wherein the surface is a palmar surface of the hand.

### Claim 18 (depends on 15)

The machine-storage medium of claim 15, wherein the operations further comprise:
measuring, using the tracking data, a distance between a first landmark on the hand and a second landmark on the hand; and
adjusting a size of the interactive virtual object using the distance.

### Claim 19 (depends on 18)

The machine-storage medium of claim 18, wherein the first landmark is a wrist landmark and the second landmark is a middle knuckle landmark.

### Claim 20 (depends on 15)

The machine-storage medium of claim 15, wherein the XR system is a head-wearable apparatus.
