# US 2023/0401795 A1 · Extended reality based digital assistant interactions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0401795 A1
- Document ID: US20230401795A1
- Status: Published application
- Kind: A1
- Application number: 18/202,849
- Filed: 2023-05-26
- Published: 2023-12-14
- Earliest claimed: 2022-06-10
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Lynn I. STREJA; Saurabh ADYA; Keith P. AVERY; Karan M. DARYANANI; Stephen O. LEMAY; Myra C. LUKENS; Sreeneel K. MADDIKA; Chaitanya MANNEMALA; Aswath MANOHARAN; Pedro MARI; Jay MOON; Abhishek RAWAT; Garrett L. WEINBERG
- Web page: https://patentgrep.com/patent/US20230401795A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 19/003: {Navigation within 3D models or images}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 13/00: Animation
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T 2219/2016: Rotation, translation, scaling
- G06T 2219/2021: Shape modification
- G10L 2015/223: {Execution procedure of a spoken command}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An example process includes: while displaying a portion of an extended reality (XR) environment representing a current field of view of a user: detecting a user gaze at a first object displayed in the XR environment, where the first object is persistent in the current field of view of the XR environment; in response to detecting the user gaze at the first object, expanding the first object into a list of objects including a second object representing a digital assistant; detecting a user gaze at the second object; in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a digital assistant session is initiated; receiving a first audio input from the user; and displaying a second animation of the second object indicating that the digital assistant is actively listening to the user.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with one or more sensors, cause the electronic device to:
while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:
detect, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;
in response to detecting the user gaze at the first object, expand the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;
detect, with the one or more sensors, a user gaze at the second object;
in accordance with detecting the user gaze at the second object, display a first animation of the second object indicating that a digital assistant session is initiated;
receive a first audio input from the user of the electronic device; and
display a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first animation of the second object includes displaying a change in a shape, a size, or a color of the second object.

### Claim 3 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first animation of the second object includes moving the second object away from the list of objects.

### Claim 4 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein displaying the first animation of the second object includes:
ceasing to display of the list of objects.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein displaying the first animation of the second object includes determining, based on the user gaze at the second object, that the first audio input is intended for the digital assistant.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
while displaying the list of objects, receive a hand gesture from the user, the hand gesture corresponding to a selection of the second object; and
in response to receiving the hand gesture, display the first animation of the second object indicating that a digital assistant session is initiated.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive a second audio input including a spoken trigger for initiating a digital assistant session; and
in response to receiving the second audio input, initiate the digital assistant session, including displaying the first animation of the second object.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the second animation of the second object includes expanding and shrinking a size of the second object responsive to the first audio input.

### Claim 9 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
display a virtual object in response to receiving the first audio input, the virtual object corresponding to a response, by the digital assistant, to the first audio input.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the virtual object and the second object are persistent in the current field of view of the XR environment.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the second object is persistent in the current field of view of the XR environment while the virtual object is positioned at a fixed location in the XR environment.

### Claim 12 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the virtual object and the second object are positioned at respective fixed locations in the XR environment.

### Claim 13 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the virtual object is displayed below the second object.

### Claim 14 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the virtual object is positioned within a predetermined distance from the second object in the XR environment.

### Claim 15 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive, from the user, a request to interact with the virtual object;
in response to receiving the request to interact with the virtual object, expand the virtual object into a user interface of an application corresponding to the virtual object.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein:
the request to interact with the virtual object corresponds to moving the virtual object from an initial location to a destination location; and
expanding the virtual object into the user interface includes displaying the user interface at the destination location.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein:
the second object is displayed while the user interface is displayed; and
the user can control the application using the second object by gazing at the second object or by speaking a trigger phrase.

### Claim 18 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive a request to integrate the virtual object into a second application; and
in response to receiving the request to integrate the virtual object into the second application, integrate the virtual object into the second application, including:
displaying a content of the virtual object within a user interface of the second application.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein:
while the content of the virtual object is displayed within the user interface of the second application:
the second object is displayed outside of the user interface of the second application.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein:
while the content of the virtual object is displayed within the user interface of the second application:
the second object is displayed inside of the user interface of the second application.

### Claim 21 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
in accordance with displaying the second object:
cease to display the second object a predetermined duration after displaying the virtual object; and
display the first object.

### Claim 22 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
determine whether one or more digital assistant dismissal criteria are satisfied; and
in accordance with a determination that the one or more digital assistant dismissal criteria are satisfied, cease to display the second object, wherein satisfying the digital assistant dismissal criteria includes at least one of:
determining that the user's attention is not directed to the virtual object;
determining that the user's attention is not directed to the virtual object for at least a second predetermined duration; and
determining that the user has used a gesture to interact with a user interface element displayed in the XR environment, wherein the user interface element does not correspond to the digital assistant.

### Claim 23 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
in accordance with displaying the second object:
cease to display the second object a third predetermined duration after receiving the first audio input from the user; and
display the first object.

### Claim 24 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein expanding the first object into the list of objects includes:
displaying a third animation of the second object.

### Claim 25

An electronic device, comprising:
one or more sensors;
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:
detecting, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;
in response to detecting the user gaze at the first object, expanding the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;
detecting, with the one or more sensors, a user gaze at the second object;
in accordance with detecting the user gaze at the second object,
displaying a first animation of the second object indicating that a digital assistant session is initiated;
receiving a first audio input from the user of the electronic device; and
displaying a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

### Claim 26

A method, comprising:
at an electronic device with one or more processors, memory, and one or more sensors:
while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:
detecting, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;
in response to detecting the user gaze at the first object, expanding the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;
detecting, with the one or more sensors, a user gaze at the second object;
in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a digital assistant session is initiated;
receiving a first audio input from the user of the electronic device; and
displaying a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.
