# US 2026/0087850 A1 · Systems and methods of processing based on user queries and gaze

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087850 A1
- Document ID: US20260087850A1
- Status: Published application
- Kind: A1
- Application number: 19/330,484
- Filed: 2025-09-16
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: William D. LINDMEIER; Devin W. CHALMERS; Sean B. KELLY
- Web page: https://patentgrep.com/patent/US20260087850A1

## CPC

- G06V 40/193: {Preprocessing; Feature extraction}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06V 10/273: {removing elements interfering with the pattern to be recognised}
- G06V 20/20: in augmented reality scenes
- G06V 20/68: Food, e.g. fruit or vegetables
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In some examples, an electronic device in communication with one or more input devices detects an input and a gaze direction of a user of the electronic device. In some examples, in response to the input, the electronic device captures one or more images. In some examples, using the detected gaze direction and a portion of the input, the electronic device identifies a subset of at least a first image from the captured images. If certain criteria are satisfied, the electronic device performs an operation using processing circuitry based on processing the input, the captured images, and the identified subset of the first image.

## Claims

### Claim 1

A method comprising:
an electronic device in communication with one or more one or more input devices:
detecting, via the one or more input devices, an input;
detecting, via the one or more input devices, a gaze direction of a user;
capturing, via the one or more input devices, one or more images;
identifying, using the gaze direction and a portion of the input, a subset of at least a first image of the one or more images; and
in accordance with a determination that one or more criteria are satisfied, performing, via processing circuitry, an operation based on processing the input, the one or more images, and the subset of the first image.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the subset of the at least the first image of the one or more images comprises:
cropping, via the processing circuitry, the subset of the first image from the first image;
identifying a predetermined region around the gaze direction of the user; or
identifying a region around the gaze direction of the user, wherein dimensions of the region around the gaze direction is based on a distance of the user from one or more objects at a focal point of the gaze direction of the user.

### Claim 3 (depends on 1)

The method of claim 1, wherein the operation includes causing a secondary electronic device in communication with the electronic device to:
output, via one or more output devices of the secondary electronic device, information related to one or more objects included in the subset of the first image; or
initiate an application based on the one or more objects included in the subset of the first image.

### Claim 4 (depends on 1)

The method of claim 1, wherein identifying the subset of the at least the first image is based on an image segmentation model, the input, and the gaze direction.

### Claim 5 (depends on 1)

The method of claim 1, wherein capturing or the first image is selected based on detecting a demonstrative pronoun in the input, wherein the input includes an audio input that includes a language command.

### Claim 6 (depends on 1)

The method of claim 1, wherein performing the operation comprises:
in accordance with identifying a first subset of the at least first image, performing a first operation based on one or more first objects in the first subset of the at least first image and the input; and
in accordance with identifying a second subset of the at least first image, different from the first subset, performing a second operation, different than the first operation, based on one or more second objects, different than the one or more first objects, in the second subset of the at least first image and the input.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the one or more images, the subset of the first image, and the input are provided to a model stored at a secondary electronic device accepting one or more image inputs and one or more language inputs; and
the method further comprises:
transmitting the input, the one or more images, and the subset of the first image to the secondary electronic device; and
receiving an output of the model from the secondary electronic device.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more criteria include a criterion that is satisfied when processing the input, the one or more images, and the subset of the first image provides a request corresponding to one or more objects within the subset of the first image.

### Claim 9

An electronic device comprising:
one or more processors;
memory; and
one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
detecting, via one or more input devices, an input;
detecting, via the one or more input devices, a gaze direction of a user;
capturing, via the one or more input devices, one or more images;
identifying, using the gaze direction and a portion of the input, a subset of at least a first image of the one or more images; and
in accordance with a determination that one or more criteria are satisfied, performing, via processing circuitry, an operation based on processing the input, the one or more images, and the subset of the first image.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein identifying the subset of the at least the first image of the one or more images comprises:
cropping, via the processing circuitry, the subset of the first image from the first image;
identifying a predetermined region around the gaze direction of the user; or
identifying a region around the gaze direction of the user, wherein dimensions of the region around the gaze direction is based on a distance of the user from one or more objects at a focal point of the gaze direction of the user.

### Claim 11 (depends on 9)

The electronic device of claim 9, wherein the operation includes causing a secondary electronic device in communication with the electronic device to:
output, via one or more output devices of the secondary electronic device, information related to one or more objects included in the subset of the first image; or
initiate an application based on the one or more objects included in the subset of the first image.

### Claim 12 (depends on 9)

The electronic device of claim 9, wherein identifying the subset of the at least the first image is based on an image segmentation model, the input, and the gaze direction.

### Claim 13 (depends on 9)

The electronic device of claim 9, wherein capturing or the first image is selected based on detecting a demonstrative pronoun in the input, wherein the input includes an audio input that includes a language command.

### Claim 14 (depends on 9)

The electronic device of claim 9, wherein performing the operation comprises:
in accordance with identifying a first subset of the at least first image, performing a first operation based on one or more first objects in the first subset of the at least first image and the input; and
in accordance with identifying a second subset of the at least first image, different from the first subset, performing a second operation, different than the first operation, based on one or more second objects, different than the one or more first objects, in the second subset of the at least first image and the input.

### Claim 15 (depends on 9)

The electronic device of claim 9, wherein:
the one or more images, the subset of the first image, and the input are provided to a model stored at a secondary electronic device accepting one or more image inputs and one or more language inputs; and
the one or more programs further include instructions for:
transmitting the input, the one or more images, and the subset of the first image to the secondary electronic device; and
receiving an output of the model from the secondary electronic device.

### Claim 16 (depends on 9)

The electronic device of claim 9, wherein the one or more criteria include a criterion that is satisfied when processing the input, the one or more images, and the subset of the first image provides a request corresponding to one or more objects within the subset of the first image.

### Claim 17

A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:
detect, via one or more input devices, an input;
detect, via the one or more input devices, a gaze direction of a user;
capture, via the one or more input devices, one or more images;
identify, using the gaze direction and a portion of the input, a subset of at least a first image of the one or more images; and
in accordance with a determination that one or more criteria are satisfied, perform, via processing circuitry, an operation based on processing the input, the one or more images, and the subset of the first image.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein identifying the subset of the at least the first image of the one or more images comprises:
cropping, via the processing circuitry, the subset of the first image from the first image;
identifying a predetermined region around the gaze direction of the user; or
identifying a region around the gaze direction of the user, wherein dimensions of the region around the gaze direction is based on a distance of the user from one or more objects at a focal point of the gaze direction of the user.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the operation includes causing a secondary electronic device in communication with the electronic device to:
output, via one or more output devices of the secondary electronic device, information related to one or more objects included in the subset of the first image; or
initiate an application based on the one or more objects included in the subset of the first image.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein identifying the subset of the at least the first image is based on an image segmentation model, the input, and the gaze direction.

### Claim 21 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein capturing or the first image is selected based on detecting a demonstrative pronoun in the input, wherein the input includes an audio input that includes a language command.

### Claim 22 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein performing the operation comprises:
in accordance with identifying a first subset of the at least first image, performing a first operation based on one or more first objects in the first subset of the at least first image and the input; and
in accordance with identifying a second subset of the at least first image, different from the first subset, performing a second operation, different than the first operation, based on one or more second objects, different than the one or more first objects, in the second subset of the at least first image and the input.

### Claim 23 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein:
the one or more images, the subset of the first image, and the input are provided to a model stored at a secondary electronic device accepting one or more image inputs and one or more language inputs; and
the instructions, when executed by the one or more processors, further cause the electronic device to:
transmit the input, the one or more images, and the subset of the first image to the secondary electronic device; and
receive an output of the model from the secondary electronic device.

### Claim 24 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the one or more criteria include a criterion that is satisfied when processing the input, the one or more images, and the subset of the first image provides a request corresponding to one or more objects within the subset of the first image.
