# US 2026/0093322 A1 · Hand and Gaze-Based Interactions with an External Device's Display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0093322 A1
- Document ID: US20260093322A1
- Status: Published application
- Kind: A1
- Application number: 19/333,996
- Filed: 2025-09-19
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Nathan Jean Emmanuel Biette; Damien Dalla-Rosa; Vincent Chapdelaine-Couture
- Web page: https://patentgrep.com/patent/US20260093322A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- B60K 35/81: for controlling displays
- G02B 27/017: {Head mounted}
- G02B 27/0176: {characterised by mechanical features}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/16: Sound input; Sound output (speech processing G10L)
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06T 19/003: {Navigation within 3D models or images}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- H04N 13/344: with head-mounted left-right displays
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Various implementations disclosed herein include devices, systems, and methods in which a head-mounted device (HMD) tracks a user's gaze and hand to augment the input capabilities of a detected display of another device, e.g., a laptop, TV, automobile device, etc. In some implementations, sensor data captured via a wearable device is used to identify a position of another device's display and/or recognize gaze and/or hand activity to identify interactions associated with content displayed on that other device's display.

## Claims

### Claim 1

A method comprising:
at a head-mounted device (HMD) comprising a processor and one or more sensors:
determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
identifying a portion of the content based on the gaze direction and the 3D position of the display;
tracking a hand of the user based on sensor data obtained via the one or more sensors; and
providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.

### Claim 2 (depends on 1)

The method of claim 1, wherein the information providing to the second device comprises a display position identified based on the gaze direction.

### Claim 3 (depends on 1)

The method of claim 1 further comprising receiving user interface element information identifying interactable objects from the second device, and the information provided to second device identifies an object of the interactable object with which the gaze direction corresponds.

### Claim 4 (depends on 1)

The method of claim 1, wherein the information provided to the second device comprises an interaction corresponding to a click or touch event on the display.

### Claim 5 (depends on 1)

The method of claim 1, wherein the second device is mobile phone or display embedded in an automobile dashboard.

### Claim 6 (depends on 1)

The method of claim 1, wherein hover-based feedback is provided by the second device based on the gaze direction.

### Claim 7 (depends on 1)

The method of claim 1, wherein hover-based feedback is provided by the HMD based on the gaze direction, wherein the HMD overlays the hover-based feedback on top of the display's content in a view provided to the user.

### Claim 8 (depends on 1)

The method of claim 1, wherein, following hover-based feedback on an interactable object, a selection type input is recognized for the interactable object based on an event identified based on the hand tracking.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining the 3D position comprises determining a location of a boundary around a content area of the display.

### Claim 10 (depends on 1)

The method of claim 1 further comprising determining to enter a second device interaction mode based on determining an intention to interact with the second device.

### Claim 11 (depends on 1)

The method of claim 1, wherein a graphical or audible indication is provided when the HMD enters a second device interaction mode.

### Claim 12 (depends on 1)

The method of claim 1, wherein input recognized based on the gaze direction is used to control the position of a cursor on the display of the second device and the tracking the hand is used to initiate interaction events based on position of the cursor.

### Claim 13 (depends on 12)

The method of claim 12, wherein control of the position of the cursor and initiation of the interaction events occurs while fingers of the hand remain above a key region of a keyboard of the second device.

### Claim 14 (depends on 13)

The method of claim 13, wherein the interaction events are initiated based on a tap on either hand of the user or a tap on the keyboard.

### Claim 15 (depends on 1)

The method of claim 1, wherein the display is a display on a dashboard of an automobile, and interaction events are initiated based on a tap on a steering wheel of the automobile while the user grips the steering wheel.

### Claim 16 (depends on 1)

The method of claim 1, wherein the content is displayed on the display of the second device and input to the second device is determined based on the user looking at the second display.

### Claim 17 (depends on 1)

The method of claim 1, wherein the content is displayed via a heads-up display and input to the second device is determined based on the user looking at the content displayed via the heads-up display.

### Claim 18 (depends on 1)

The method of claim 1, wherein the content is displayed by the HMD in a way that mimic a heads-up display and input to the second device is determined based on the user looking at the content displayed by the HMD.

### Claim 19

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
identifying a portion of the content based on the gaze direction and the 3D position of the display;
tracking a hand of the user based on sensor data obtained via the one or more sensors; and
providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.

### Claim 20

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
determining a three-dimensional position of a display of a second device, wherein content originating from the second device is visible to a user of the HMD;
determining a gaze direction of the user based on sensor data obtained via the one or more sensors;
identifying a portion of the content based on the gaze direction and the 3D position of the display;
tracking a hand of the user based on sensor data obtained via the one or more sensors; and
providing information to the second device based on identifying the portion of the content and the hand tracking, wherein the second device detects input associated with the content based on the information.
