# US 12,001,933 B2 · Virtual assistant in a communication session

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,933 B2
- Document ID: US12001933B2
- Status: Granted
- Kind: B2
- Application number: 17/949,512
- Filed: 2022-09-21
- Published: 2023-01-19
- Granted: 2024-06-04
- Earliest claimed: 2015-05-15
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Mehul K. Sanghavi; Jeffrey P. Schwerdtfeger
- Web page: https://patentgrep.com/patent/US12001933B2

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06Q 10/10: Office automation; Time management
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/212: using filtering or selective blocking
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Disclosed are techniques for utilizing a virtual assistant as part of a communication session. One or more of the participant users can select to utilize a virtual assistant to assist the participant users with tasks during the communication session. A user can use a communication application to enter a message directed to the virtual assistant. The virtual assistant can analyze the entered message and determine that the message was directed to the virtual assistant. As a result, the virtual assistant will assist the user with the identified task. A virtual assistant can assist a user with a variety of different tasks. For example, the virtual assistant can gather requested data for the user from one or more web servers, such as weather data, traffic data, etc.

## Claims

### Claim 1

A non-transitory computer-readable storage medium containing instructions that, when executed by a computer processor of a first client device, cause the first client device to:
display, by the first client device, as part of a communication session that takes place within a messaging application between a first user operating the first client device and a second user operating a second client device, a first message prepared by the first user;
determine, by a virtual assistant of the first client device, that the first message identifies a first task with which the virtual assistant can assist the first user;
in response to determining that the first message identifies the first task, assist, by the first client device, the first user with the first task, wherein assisting the first user with the first task includes:
displaying, as part of the communication session and within the messaging application, a response generated by the virtual assistant based on the first task, wherein the response is not transmitted to the second client device;
receive a user selection of the displayed response;
in response to receiving the user selection of the displayed response, present, on the first client device and as part of the communication session, additional content generated by the virtual assistant based on the displayed response;
while presenting the additional content, receive a user selection of the additional content; and
in accordance with receiving the user selection of the additional content:
transmit, to the second client device, a second message associated with the additional content.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by the computer processor, further cause the first client device to:
display, by the first client device, as part of the communication session, a third message prepared by the first user;
determine, by the virtual assistant of the first client device, based on a semantic analysis of the third message, whether the third message is directed to the virtual assistant or to the second user;
determine, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message is directed to the second user; and
in response to determining that the third message is directed to the second user, transmit, by the first client device to the second client device, the third message as part of the communication session.

### Claim 3 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the instructions, when executed by the computer processor, further cause the first client device to:
determine, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message identifies a second task with which the virtual assistant can assist the first user; and
in response to determining that the third message identifies the second task, assist, by the first client device, the first user with the second task.

### Claim 4 (depends on 3)

The non-transitory computer-readable storage medium of claim 3, wherein the instructions, when executed by the computer processor, further cause the first client device to:
present, on the first client device and as part of the communication session, a request message asking the first user whether the first user would like the virtual assistant to assist the first user with the second task; and
receive an input indicating that the first user would like the virtual assistant to assist the first user with the second task.

### Claim 5 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the displayed response includes one or more words determined based on the first task.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, by the virtual assistant of the first client device, a plurality of applications installed on the first client device that are capable of completing the first task, wherein:
the displayed response prompts the first user to select an application from the plurality of applications to complete the first task; and
the user selection of the displayed response selects a first application from the plurality of applications;
in accordance with receiving user selection of the displayed response, execute the selected first application to perform the first task; and
present, on the first client device and as part of the communication session, a confirmation message indicating the first task was performed.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first task comprises retrieving information associated with the first message.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by the computer processor, further cause the first client device to:
transmit, to the second client device, and as part of the communication session, a notification message indicating that the virtual assistant is being utilized in relation to the communication session.

### Claim 9 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first task comprises performing a financial transaction between the first user and the second user.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, by the virtual assistant of the first client device, a set of financial applications installed on both the first client device and the second client device that are capable of completing the financial transaction;
transmit, to the second client device, a third message as part of the communication session, the third message prompting the second user to select a financial application from the set of financial applications to complete the financial transaction;
receive, from the second client device, a first reply message identifying the financial application selected from the set of financial applications selected by the second user to complete the financial transaction; and
execute the financial application.

### Claim 11 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first task comprises scheduling a meeting including the first user and the second user.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, based on calendar data for the first user and the second user, a set of candidate times for the meeting;
place a temporary hold on the set of candidate times, wherein the set of candidate times appear booked in calendars of the first user and the second user while the temporary hold is in place;
present the first user and the second user with a poll to select from the set of candidate times, the poll presented as part of the communication session; and
schedule the meeting during one of the candidate times based on response data received from the first user and the second user to the poll.

### Claim 13 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the first task comprises determining an estimated time of arrival for the second user.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein determining the estimated time of arrival for the second user comprises:
requesting authorization to access location information of the second user; and
in accordance with a determination that the second user selects to authorize access to the location information of the second user:
gathering the location information of the second user; and
displaying, based on the location information of the second user, the estimated time of arrival the second user.

### Claim 15 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the virtual assistant of the first client device determines, based on a semantic analysis of the first message, that the first message identifies the first task with which the virtual assistant can assist the first user.

### Claim 16 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the displayed response corresponds to a suggestion generated by the virtual assistant of the first client device.

### Claim 17 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the user selection of the additional content authorizes information to be sent to the second client device, wherein the information corresponds to the first task.

### Claim 18 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed by the computer processor, further cause the first client device to:
after transmitting, to the second client device, the second message:
receive, from the second client device, a response to the second message; and
display, as part of the communication session, the response to the second message.

### Claim 19

A method, comprising:
at a first client device with one or more processors, memory, and a display:
displaying, by the first client device, as part of a communication session that takes place within a messaging application between a first user operating the first client device and a second user operating a second client device, a first message prepared by the first user;
determining, by a virtual assistant of the first client device, that the first message identifies a first task with which the virtual assistant can assist the first user;
in response to determining that the first message identifies the first task, assisting, by the first client device, the first user with the first task, wherein assisting the first user with the first task includes:
displaying, as part of the communication session and within the messaging application, a response generated by the virtual assistant based on the first task, wherein the response is not transmitted to the second client device;
receiving a user selection of the displayed response;
in response to receiving the user selection of the displayed response, presenting, on the first client device and as part of the communication session, additional content generated by the virtual assistant based on the displayed response;
while presenting the additional content, receiving a user selection of the additional content; and
in accordance with receiving the user selection of the additional content:
transmitting, to the second client device, a second message associated with the additional content.

### Claim 20

A first client device, comprising:
a computer processor; and
a memory containing instructions that, when executed by the computer processor, cause the first client device to:
display, by the first client device, as part of a communication session that takes place within a messaging application between a first user operating the first client device and a second user operating a second client device, a first message prepared by the first user;
determine, by a virtual assistant of the first client device, that the first message identifies a first task with which the virtual assistant can assist the first user;
in response to determining that the first message identifies the first task, assist, by the first client device, the first user with the first task, wherein assisting the first user with the first task includes:
displaying, as part of the communication session and within the messaging application, a response generated by the virtual assistant based on the first task, wherein the response is not transmitted to the second client device;
receive a user selection of the displayed response;
in response to receiving the user selection of the displayed response, present, on the first client device and as part of the communication session, additional content generated by the virtual assistant based on the displayed response;
while presenting the additional content, receive a user selection of the additional content; and
in accordance with receiving the user selection of the additional content: transmit, to the second client device, a second message associated with the additional content.

### Claim 21 (depends on 19)

The method of claim 19, further comprising:
displaying, by the first client device, as part of the communication session, a third message prepared by the first user;
determining, by the virtual assistant of the first client device, based on a semantic analysis of the third message, whether the third message is directed to the virtual assistant or to the second user;
determining, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message is directed to the second user; and
in response to determining that the third message is directed to the second user, transmitting, by the first client device to the second client device, the third message as part of the communication session.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
determining, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message identifies a second task with which the virtual assistant can assist the first user; and
in response to determining that the third message identifies the second task, assisting, by the first client device, the first user with the second task.

### Claim 23 (depends on 22)

The method of claim 22, further comprising:
presenting, on the first client device and as part of the communication session, a request message asking the first user whether the first user would like the virtual assistant to assist the first user with the second task; and
receiving an input indicating that the first user would like the virtual assistant to assist the first user with the second task.

### Claim 24 (depends on 19)

The method of claim 19, wherein the displayed response includes one or more words determined based on the first task.

### Claim 25 (depends on 19)

The method of claim 19, further comprising:
identifying, by the virtual assistant of the first client device, a plurality of applications installed on the first client device that are capable of completing the first task, wherein:
the displayed response prompts the first user to select an application from the plurality of applications to complete the first task; and
the user selection of the displayed response selects a first application from the plurality of applications;
in accordance with receiving user selection of the displayed response, executing the selected first application to perform the first task; and
presenting, on the first client device and as part of the communication session, a confirmation message indicating the first task was performed.

### Claim 26 (depends on 19)

The method of claim 19, wherein the first task comprises retrieving information associated with the first message.

### Claim 27 (depends on 19)

The method of claim 19, further comprising:
transmitting, to the second client device, and as part of the communication session, a notification message indicating that the virtual assistant is being utilized in relation to the communication session.

### Claim 28 (depends on 19)

The method of claim 19, wherein the first task comprises performing a financial transaction between the first user and the second user.

### Claim 29 (depends on 28)

The method of claim 28, further comprising:
identifying, by the virtual assistant of the first client device, a set of financial applications installed on both the first client device and the second client device that are capable of completing the financial transaction;
transmitting, to the second client device, a third message as part of the communication session, the third message prompting the second user to select a financial application from the set of financial applications to complete the financial transaction;
receiving, from the second client device, a first reply message identifying the financial application selected from the set of financial applications selected by the second user to complete the financial transaction; and
executing the financial application.

### Claim 30 (depends on 19)

The method of claim 19, wherein the first task comprises scheduling a meeting including the first user and the second user.

### Claim 31 (depends on 30)

The method of claim 30, further comprising:
identifying, based on calendar data for the first user and the second user, a set of candidate times for the meeting;
placing a temporary hold on the set of candidate times, wherein the set of candidate times appear booked in calendars of the first user and the second user while the temporary hold is in place;
presenting the first user and the second user with a poll to select from the set of candidate times, the poll presented as part of the communication session; and
scheduling the meeting during one of the candidate times based on response data received from the first user and the second user to the poll.

### Claim 32 (depends on 19)

The method of claim 19, wherein the first task comprises determining an estimated time of arrival for the second user.

### Claim 33 (depends on 32)

The method of claim 32, wherein determining the estimated time of arrival for the second user comprises:
requesting authorization to access location information of the second user; and
in accordance with a determination that the second user selects to authorize access to the location information of the second user:
gathering the location information of the second user; and
displaying, based on the location information of the second user, the estimated time of arrival the second user.

### Claim 34 (depends on 19)

The method of claim 19, wherein the virtual assistant of the first client device determines, based on a semantic analysis of the first message, that the first message identifies the first task with which the virtual assistant can assist the first user.

### Claim 35 (depends on 19)

The method of claim 19, wherein the displayed response corresponds to a suggestion generated by the virtual assistant of the first client device.

### Claim 36 (depends on 19)

The method of claim 19, wherein the user selection of the additional content authorizes information to be sent to the second client device, wherein the information corresponds to the first task.

### Claim 37 (depends on 19)

The method of claim 19, further comprising:
after transmitting, to the second client device, the second message:
receiving, from the second client device, a response to the second message; and
displaying, as part of the communication session, the response to the second message.

### Claim 38 (depends on 20)

The first client device of claim 20, wherein the instructions, when executed by the computer processor, further cause the first client device to:
display, by the first client device, as part of the communication session, a third message prepared by the first user;
determine, by the virtual assistant of the first client device, based on a semantic analysis of the third message, whether the third message is directed to the virtual assistant or to the second user;
determine, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message is directed to the second user; and
in response to determining that the third message is directed to the second user, transmit, by the first client device to the second client device, the third message as part of the communication session.

### Claim 39 (depends on 38)

The first client device of claim 38, wherein the instructions, when executed by the computer processor, further cause the first client device to:
determine, by the virtual assistant of the first client device, based on the semantic analysis of the third message, that the third message identifies a second task with which the virtual assistant can assist the first user; and
in response to determining that the third message identifies the second task, assist, by the first client device, the first user with the second task.

### Claim 40 (depends on 39)

The first client device of claim 39, wherein the instructions, when executed by the computer processor, further cause the first client device to:
present, on the first client device and as part of the communication session, a request message asking the first user whether the first user would like the virtual assistant to assist the first user with the second task; and
receive an input indicating that the first user would like the virtual assistant to assist the first user with the second task.

### Claim 41 (depends on 20)

The first client device of claim 20, wherein the displayed response includes one or more words determined based on the first task.

### Claim 42 (depends on 20)

The first client device of claim 20, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, by the virtual assistant of the first client device, a plurality of applications installed on the first client device that are capable of completing the first task, wherein:
the displayed response prompts the first user to select an application from the plurality of applications to complete the first task; and
the user selection of the displayed response selects a first application from the plurality of applications;
in accordance with receiving user selection of the displayed response, execute the selected first application to perform the first task; and
present, on the first client device and as part of the communication session, a confirmation message indicating the first task was performed.

### Claim 43 (depends on 20)

The first client device of claim 20, wherein the first task comprises retrieving information associated with the first message.

### Claim 44 (depends on 20)

The first client device of claim 20, wherein the instructions, when executed by the computer processor, further cause the first client device to:
transmit, to the second client device, and as part of the communication session, a notification message indicating that the virtual assistant is being utilized in relation to the communication session.

### Claim 45 (depends on 20)

The first client device of claim 20, wherein the first task comprises performing a financial transaction between the first user and the second user.

### Claim 46 (depends on 45)

The first client device of claim 45, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, by the virtual assistant of the first client device, a set of financial applications installed on both the first client device and the second client device that are capable of completing the financial transaction;
transmit, to the second client device, a third message as part of the communication session, the third message prompting the second user to select a financial application from the set of financial applications to complete the financial transaction;
receive, from the second client device, a first reply message identifying the financial application selected from the set of financial applications selected by the second user to complete the financial transaction; and
execute the financial application.

### Claim 47 (depends on 20)

The first client device of claim 20, wherein the first task comprises scheduling a meeting including the first user and the second user.

### Claim 48 (depends on 47)

The first client device of claim 47, wherein the instructions, when executed by the computer processor, further cause the first client device to:
identify, based on calendar data for the first user and the second user, a set of candidate times for the meeting;
place a temporary hold on the set of candidate times, wherein the set of candidate times appear booked in calendars of the first user and the second user while the temporary hold is in place;
present the first user and the second user with a poll to select from the set of candidate times, the poll presented as part of the communication session; and
schedule the meeting during one of the candidate times based on response data received from the first user and the second user to the poll.

### Claim 49 (depends on 20)

The first client device of claim 20, wherein the first task comprises determining an estimated time of arrival for the second user.

### Claim 50 (depends on 49)

The first client device of claim 49, wherein determining the estimated time of arrival for the second user comprises:
requesting authorization to access location information of the second user; and
in accordance with a determination that the second user selects to authorize access to the location information of the second user:
gathering the location information of the second user; and
displaying, based on the location information of the second user, the estimated time of arrival the second user.

### Claim 51 (depends on 20)

The first client device of claim 20, wherein the virtual assistant of the first client device determines, based on a semantic analysis of the first message, that the first message identifies the first task with which the virtual assistant can assist the first user.

### Claim 52 (depends on 20)

The first client device of claim 20, wherein the displayed response corresponds to a suggestion generated by the virtual assistant of the first client device.

### Claim 53 (depends on 20)

The first client device of claim 20, wherein the user selection of the additional content authorizes information to be sent to the second client device, wherein the information corresponds to the first task.

### Claim 54 (depends on 20)

The first client device of claim 20, wherein the instructions, when executed by the computer processor, further cause the first client device to:
after transmitting, to the second client device, the second message:
receive, from the second client device, a response to the second message; and
display, as part of the communication session, the response to the second message.