# US 2024/0256052 A1 · User interactions with remote devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256052 A1
- Document ID: US20240256052A1
- Status: Published application
- Kind: A1
- Application number: 18/587,721
- Filed: 2024-02-26
- Published: 2024-08-01
- Earliest claimed: 2021-10-12
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Jonathan KIES; Leonid SHEYNBLAT
- Web page: https://patentgrep.com/patent/US20240256052A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 40/18: Eye characteristics, e.g. of the iris
- G08C 17/02: using a radio link
- G08C 23/04: using light waves, e.g. infrared
- G06F 2218/12: Classification; Matching
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08C: TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS (fluid pressure transmitting systems F15B; mechanical means for transferring the output of a sensing member into a different variable G01D5/00; mechanical control systems G05G)

## Abstract

Systems, methods, and non-transitory media are provided for presenting information associated with at least one input option. An example method can include receiving data identifying one or more input options associated with a first device in a scene; determining, including using at least one memory, information relevant to at least one of the scene, the first device, and a user associated with a second device; and based on the one or more input options and the information, output user guidance data corresponding to an input option for which relevant context information has been determined.

## Claims

### Claim 1

An extended reality head-mounted apparatus for outputting information associated with at least one input option, comprising:
a display;
at least one sensor;
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
identify a device in a scene within a field-of-view of the at least one sensor;
receive data identifying a plurality of input options associated with the device in the scene;
determine context information relevant to at least one of the scene, the device, or a user associated with the extended reality head-mounted apparatus; and
output user guidance data via the display, the user guidance data corresponding to an interaction with the device using one or more input options from the plurality of input options associated with the device, wherein the one or more input options are associated with the context information.

### Claim 2 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein, to output the user guidance data, the at least one processor is configured to output, via the display, at least one of a user input element associated with the one or more input options, a virtual overlay on a physical object associated with the one or more input options, or a cue indicating how to provide an input associated with the one or more input options.

### Claim 3 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
based on the context information, predict a user interaction with the device; and
output the user guidance data corresponding to the interaction with the device using the one or more input options based on the plurality of input options and the predicted user interaction.

### Claim 4 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the device comprises a connected device with network communication capabilities, and the at least one processor is configured to:
based on the context information and the plurality of input options, determine a hand gesture that represents a predicted user interaction, the predicted user interaction comprising a predicted user input to the device, wherein the user guidance data comprises an indication of the hand gesture that, when detected, invokes an actual user input at the device.

### Claim 5 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein, to output the user guidance data, the at least one processor is configured to:
render, via the display, a virtual overlay configured to appear to be located on a surface of the device, the virtual overlay comprising a user interface element associated with the one or more input options, wherein the user interface element comprises at least one of a virtual user input object associated with the one or more input options or a visual indication of a physical control object on the device configured to receive input corresponding to the one or more input options.

### Claim 6 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the context information comprises at least one of an eye gaze of the user or a pose of the user, the at least one processor being configured to:
predict a user interaction with the device based on at least one of the eye gaze of the user or the pose of the user;
based on outputting the user guidance data via the display, detect an actual user input associated with the one or more input options, the actual user input representing the predicted user interaction; and
transmit, to the device, a command corresponding to the actual user input associated with the one or more input options.

### Claim 7 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein, to output the user guidance data corresponding to the interaction with the device using the one or more input options, the at least one processor is configured to:
display the user guidance data via the display.

### Claim 8 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein, to output the user guidance data corresponding to the interaction with the device using the one or more input options, the at least one processor is configured to:
output audio data representing the user guidance data.

### Claim 9 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein, to output the user guidance data corresponding to the interaction with the device using the one or more input options, the at least one processor is configured to:
display the user guidance data via the display; and
output audio data associated with the displayed user guidance data.

### Claim 10 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
receive, from the device, the data identifying the plurality of input options associated with the device.

### Claim 11 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
receive, from a server, the data identifying the plurality of input options associated with the device.

### Claim 12 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the device has no external user interface for receiving one or more user inputs.

### Claim 13 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
based on the context information, refrain from outputting additional user guidance data associated with the device.

### Claim 14 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
obtain a user input associated with an input option of the one or more input options; and
output, to the device, an instruction corresponding to the user input, the instruction being configured to control one or more operations of the device according to the input option.

### Claim 15 (depends on 14)

The extended reality head-mounted apparatus of claim 14, wherein the user input is a gesture input.

### Claim 16 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the at least one processor is configured to:
establish a connection with the device.

### Claim 17 (depends on 16)

The extended reality head-mounted apparatus of claim 16, wherein, to establish the connection with the device, the at least one processor is configured to pair with the device.

### Claim 18 (depends on 1)

The extended reality head-mounted apparatus of claim 1, wherein the context information relevant to the device comprises a power setting of the device.

### Claim 19

A method for outputting information at an extended reality head-mounted device, the method comprising:
identifying a device in a scene within a field-of-view of at least one sensor of the extended reality head-mounted device;
receiving, at the extended reality head-mounted device, data identifying a plurality of input options associated with the device in the scene;
determining, at the extended reality head-mounted device, context information relevant to at least one of the scene, the device, or a user associated with head-mounted device; and
outputting user guidance data via a display of the extended reality head-mounted device, the user guidance data corresponding to an interaction with the device using one or more input options from the plurality of input options associated with the device, wherein the one or more input options are associated with the context information.

### Claim 20 (depends on 19)

The method of claim 19, wherein the user guidance data output via the display comprises at least one of a user input element associated with the one or more input options, a virtual overlay on a physical object associated with the one or more input options, or a cue indicating how to provide an input associated with the one or more input options.

### Claim 21 (depends on 19)

The method of claim 19, further comprising:
based on the context information, predicting a user interaction with the device; and
outputting the user guidance data corresponding to interaction with the device using the one or more input options based on the plurality of input options and the predicted user interaction.

### Claim 22 (depends on 21)

The method of claim 21, wherein the device comprises a connected device with network communication capabilities, and the method further comprises:
based on the context information and the plurality of input options, determining a hand gesture that represents a predicted user interaction, the predicted user interaction comprising a predicted user input to the device, wherein the user guidance data comprises an indication of the hand gesture that, when detected, invokes an actual user input at the device.

### Claim 23 (depends on 19)

The method of claim 19, wherein the context information comprises at least one of an eye gaze of the user or a pose of the user, the method further comprising:
predicting a user interaction with the device based on at least one of the eye gaze of the user or the pose of the user;
based on outputting the user guidance data via the display, detecting an actual user input associated with the one or more input options, the actual user input representing the predicted user interaction; and
transmitting, to the device, a command corresponding to the actual user input associated with the one or more input options.

### Claim 24 (depends on 19)

The method of claim 19, wherein outputting the user guidance data corresponding to interaction with the device using the one or more input options comprises:
displaying the user guidance data via the display.

### Claim 25 (depends on 19)

The method of claim 19, wherein outputting the user guidance data corresponding to interaction with the device using the one or more input options comprises:
outputting audio data representing the user guidance data.

### Claim 26 (depends on 19)

The method of claim 19, further comprising:
receiving, from the device, the data identifying the plurality of input options associated with the device.

### Claim 27 (depends on 19)

The method of claim 19, further comprising:
receiving, from a server, the data identifying the plurality of input options associated with the device.

### Claim 28 (depends on 19)

The method of claim 19, wherein the device has no external user interface for receiving one or more user inputs.

### Claim 29 (depends on 19)

The method of claim 19, further comprising:
obtaining a user input associated with an input option of the one or more input options; and
outputting, to the device, an instruction corresponding to the user input, the instruction being configured to control one or more operations of the device according to the input option.

### Claim 30 (depends on 29)

The method of claim 29, wherein the user input is a gesture input.
