# US 2024/0393876 A1 · User interface response based on gaze-holding event assessment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0393876 A1
- Document ID: US20240393876A1
- Status: Published application
- Kind: A1
- Application number: 18/790,194
- Filed: 2024-07-31
- Published: 2024-11-28
- Earliest claimed: 2022-09-22
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Vinay Chawda; Mehmet N. Agaoglu; Leah M. Gum; Paul A. Lacey; Julian K. Shutzberg; Tim H. Cornelissen; Alexander G. Berardino
- Web page: https://patentgrep.com/patent/US20240393876A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various implementations provide views of 3D environments (e.g., extended reality (XR) environments). Non-eye-based user activity, such as hand gestures, is associated with some types of eye-based activity, such as the user gazing at a particular user interface component displayed within a view of a 3D environment. For example, a user's pinching hand gesture may be associated with the user gazing at a particular user interface component, such as a button, at around the same time as the pinching hand gesture is made. These associated behaviors (e.g., the pinch and gaze at the button) may then be interpreted as user input, e.g., user input selecting or otherwise acting upon that user interface component. In some implementations, non-eye-based user activity is only associated with types of eye-based user activity that are likely to correspond to a user perceiving what they are seeing and/or intentionally looking at something.

## Claims

### Claim 1

A method comprising:
at an electronic device having a processor and one or more sensors:
obtaining gaze motion classification data that was generated based on sensor data of an eye captured by the one or more sensors, the gaze motion classification data distinguishing a first type of eye behavior occurring at a first time period from a second type of eye behavior different than the first type of eye behavior occurring at a second time period;
detecting a user activity; and
triggering a user interface response based on the user activity and the gaze motion classification data.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the user activity is a gaze in a gaze direction; and
the user interface response comprises providing an indication of user attention to a user interface element based on the gaze motion classification data.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the user activity is a gesture distinct from a gaze; and
the user interface response is triggered based on associating the user activity with a user interface element.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the user activity is an input device interaction distinct from a gaze; and
the user interface response is triggered based on associating the user activity with a user interface element.

### Claim 5 (depends on 1)

The method of claim 1 further comprising associating the user activity with a user interface element based on when the user activity occurred.

### Claim 6 (depends on 5)

The method of claim 5, wherein associating the user activity with the user interface element comprises determining that the user activity occurred within a threshold time of a time period associated with a type of eye behavior.

### Claim 7 (depends on 1)

The method of claim 1, wherein the user activity is:
a finger pinch;
a multi-finger gesture;
a hand grasping gesture;
finger pinches made by both hands;
grasping gestures made by both hands;
a gesture made by moving both hands relative to one another;
a head movement;
user input provided via a hand-held device;
user input provided via an input device; or
user input provided via a wearable device.

### Claim 8 (depends on 1)

The method of claim 1, wherein the user activity is detected based on images captured from an outward-facing camera of a head-mounted device.

### Claim 9 (depends on 1)

The method of claim 1, wherein obtaining the gaze motion classification data comprises receiving a signal of live gaze velocity data based on a stream of live images of the eye.

### Claim 10 (depends on 1)

The method of claim 1, wherein identifying the gaze-holding events comprises:
motion classification;
pose stabilization; or
blink removal.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the user activity is a saccadic event having a characteristic that exceeds a threshold; and
the user interface response is based on excluding gaze behavior occurring during one or more time periods following the saccadic event.

### Claim 12 (depends on 1)

The method of claim 1, wherein the user interface response is based on excluding gaze behavior occurring during a time period following an event based on:
(a) an amplitude representing gaze velocity change during the event;
(b) a rate of change of gaze velocity during the event;
(c) a duration of the gaze behavior; or
(d) distance of eye travel during the event.

### Claim 13

A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
obtaining gaze motion classification data that was generated based on sensor data of an eye captured by the one or more sensors, the gaze motion classification data distinguishing a first type of eye behavior occurring at a first time period from a second type of eye behavior different than the first type of eye behavior occurring at a second time period;
detecting a user activity; and
triggering a user interface response based on the user activity and the gaze motion classification data.

### Claim 14 (depends on 13)

The system of claim 13, wherein:
the user activity is a gaze in a gaze direction; and
the user interface response comprises providing an indication of user attention to a user interface element based on the gaze motion classification data.

### Claim 15 (depends on 13)

The system of claim 13, wherein:
the user activity is a gesture distinct from a gaze; and
the user interface response is triggered based on associating the user activity with a user interface element.

### Claim 16 (depends on 13)

The system of claim 13, wherein:
the user activity is an input device interaction distinct from a gaze; and
the user interface response is triggered based on associating the user activity with a user interface element.

### Claim 17 (depends on 13)

The system of claim 13, wherein the operations further comprise associating the user activity with a user interface element based on when the user activity occurred.

### Claim 18 (depends on 17)

The system of claim 17, wherein associating the user activity with the user interface element comprises determining that the user activity occurred within a threshold time of a time period associated with a type of eye behavior.

### Claim 19 (depends on 13)

The system of claim 13, wherein the user activity is:
a finger pinch;
a multi-finger gesture;
a hand grasping gesture;
finger pinches made by both hands;
grasping gestures made by both hands;
a gesture made by moving both hands relative to one another;
a head movement;
user input provided via a hand-held device;
user input provided via an input device; or
user input provided via a wearable device.

### Claim 20 (depends on 13)

The system of claim 13, wherein the user activity is detected based on images captured from an outward-facing camera of a head-mounted device.

### Claim 21 (depends on 13)

The system of claim 13, wherein obtaining the gaze motion classification data comprises receiving a signal of live gaze velocity data based on a stream of live images of the eye.

### Claim 22 (depends on 13)

The system of claim 13, wherein identifying the gaze-holding events comprises:
motion classification;
pose stabilization; or
blink removal.

### Claim 23 (depends on 13)

The system of claim 13, wherein:
the user activity is a saccadic event having a characteristic that exceeds a threshold; and
the user interface response is based on excluding gaze behavior occurring during one or more time periods following the saccadic event.

### Claim 24 (depends on 13)

The system of claim 13, wherein the user interface response is based on excluding gaze behavior occurring during a time period following an event based on:
(a) an amplitude representing gaze velocity change during the event;
(b) a rate of change of gaze velocity during the event;
(c) a duration of the gaze behavior; or
(d) distance of eye travel during the event.

### Claim 25

A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
obtaining gaze motion classification data that was generated based on sensor data of an eye captured by the one or more sensors, the gaze motion classification data distinguishing a first type of eye behavior occurring at a first time period from a second type of eye behavior different than the first type of eye behavior occurring at a second time period;
detecting a user activity; and
triggering a user interface response based on the user activity and the gaze motion classification data.
