# US 2025/0095648 A1 · Digital assistant interaction in a video communication session environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095648 A1
- Document ID: US20250095648A1
- Status: Published application
- Kind: A1
- Application number: 18/962,862
- Filed: 2024-11-27
- Published: 2025-03-20
- Earliest claimed: 2020-02-12
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Niranjan MANJUNATH; Willem MATTELAER; Jessica PECK; Lily Shuting ZHANG
- Web page: https://patentgrep.com/patent/US20250095648A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 7/147: {Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals (selecting H04Q)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 1/72469: for operating the device by selecting functions from two or more displayed items, e.g. menus or icons
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G10L 2015/223: {Execution procedure of a spoken command}
- H04M 2250/74: with voice recognition means
- H04N 2007/145: {Handheld terminals}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

This relates to an intelligent automated assistant in a video communication session environment. An example method includes, during a video communication session between at least two user devices, and at a first user device: receiving a first user voice input; in accordance with a determination that the first user voice input represents a communal digital assistant request, transmitting a request to provide context information associated with the first user voice input to the first user device; receiving context information associated with the first user voice input; obtaining a first digital assistant response based at least on a portion of the context information received from the second user device and at least a portion of context information associated with the first user voice input that is stored on the first user device; providing the first digital assistant response to the second user device; and outputting the first digital assistant response.

## Claims

### Claim 1

A first user device, comprising:
a display;
a first microphone;
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, wherein the one or more programs include instructions for:
during a communication session between at least the first user device and a second user device with a second microphone:
displaying a communication session user interface, wherein the communication session user interface includes a user interface element that represents a digital assistant;
receiving a first user input corresponding to a selection of the user interface element that represents the digital assistant;
in response to receiving the first user input corresponding to the selection of the user interface element that represents the digital assistant, invoking a digital assistant session during the communication session; and
while the digital assistant session is invoked during the communication session:
receiving, via the first microphone of the first user device, a second user input that represents a first voice communication of the communication session; and
receiving a third user input that represents a second voice communication of the communication session, wherein the third user input is initially received via the second microphone of the second user device; and
outputting a digital assistant response, wherein the digital assistant response is determined based on the first voice communication and the second voice communication.

### Claim 2 (depends on 1)

The first user device of claim 1, wherein the communication session between at least the first user device and the second user device is a video communication session.

### Claim 3 (depends on 1)

The first user device of claim 1, the one or more programs further including instructions for storing data associated with the digital assistant response, wherein the data associated with the digital assistant response is determined based on the first voice communication and the second voice communication.

### Claim 4 (depends on 3)

The first user device of claim 3, the one or more programs further including instructions for displaying an indication that the data associated with the digital assistant response is available.

### Claim 5 (depends on 3)

The first user device of claim 3, wherein the data associated with the digital assistant response is accessible by the first user device and the second user device.

### Claim 6 (depends on 1)

The first user device of claim 1, wherein outputting the digital assistant response includes displaying a first portion of the digital assistant response at the first user device during the communication session and storing a second portion of the digital assistant response that is different from the first portion of the digital assistant response.

### Claim 7 (depends on 6)

The first user device of claim 6, wherein:
the first portion of the digital assistant response corresponds to a transcription of the first voice communication and/or the second voice communication; and
the second portion of the digital assistant response corresponds to an instruction that is received during the communication session.

### Claim 8 (depends on 6)

The first user device of claim 6, the one or more programs further including instructions for:
receiving, at the first user device, a fourth user input corresponding to a request to access an application; and
in response to receiving the fourth user input, displaying, within a user interface of the application, an indication that the digital assistant response is available.

### Claim 9 (depends on 1)

The first user device of claim 1, wherein outputting the digital assistant response includes displaying text that is determined based on the first voice communication and the second voice communication.

### Claim 10 (depends on 1)

The first user device of claim 1, wherein outputting the digital assistant response includes displaying the digital assistant response during the communication session.

### Claim 11 (depends on 1)

The first user device of claim 1, wherein outputting the digital assistant response includes displaying the digital assistant response after the communication session has ended.

### Claim 12 (depends on 1)

The first user device of claim 1, wherein outputting the digital assistant response includes outputting the digital assistant response at the first user device and the second user device.

### Claim 13 (depends on 1)

The first user device of claim 1, wherein the digital assistant response is determined based on dialog history corresponding to the communication session.

### Claim 14 (depends on 1)

The first user device of claim 1, wherein the digital assistant response includes one or more candidate tasks based on the first voice communication and the second voice communication.

### Claim 15 (depends on 1)

The first user device of claim 1, the one or more programs further including instructions for:
receiving, after the communication session has ended, a fifth user input; and
in response to receiving the fifth user input, outputting the digital assistant response.

### Claim 16 (depends on 14)

The first user device of claim 14, wherein outputting the digital assistant response includes displaying the one or more candidate tasks overlaying the communication session user interface during the communication session.

### Claim 17 (depends on 14)

The first user device of claim 14, wherein outputting the digital assistant response includes displaying the one or more candidate tasks after the communication session has ended.

### Claim 18 (depends on 14)

The first user device of claim 14, wherein the one or more candidate tasks are one or more task suggestions.

### Claim 19

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of a first user device with a display and a first microphone, cause the first user device to:
during a communication session between at least the first user device and a second user device with a second microphone:
display a communication session user interface, wherein the communication session user interface includes a user interface element that represents a digital assistant;
receive a first user input corresponding to a selection of the user interface element that represents the digital assistant;
in response to receiving the first user input corresponding to the selection of the user interface element that represents the digital assistant, invoking a digital assistant session during the communication session; and
while the digital assistant session is invoked during the communication session:
receive, via the first microphone of the first user device, a second user input that represents a first voice communication of the communication session; and
receive a third user input that represents a second voice communication of the communication session, wherein the third user input is initially received via the second microphone of the second user device; and
output a digital assistant response, wherein the digital assistant response is determined based on the first voice communication and the second voice communication.

### Claim 20

A method, comprising:
at a first user device with a display and a first microphone:
during a communication session between at least the first user device and a second user device with a second microphone:
displaying a communication session user interface, wherein the communication session user interface includes a user interface element that represents a digital assistant;
receiving a first user input corresponding to a selection of the user interface element that represents the digital assistant;
in response to receiving the first user input corresponding to the selection of the user interface element that represents the digital assistant, invoking a digital assistant session during the communication session; and
while the digital assistant session is invoked during the communication session:
receiving, via the first microphone of the first user device, a second user input that represents a first voice communication of the communication session; and
receiving a third user input that represents a second voice communication of the communication session, wherein the third user input is initially received via the second microphone of the second user device; and
outputting a digital assistant response, wherein the digital assistant response is determined based on the first voice communication and the second voice communication.
