# US 2021/0264913 A1 · Voice interaction at a primary device to access call functionality of a companion device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264913 A1
- Document ID: US20210264913A1
- Status: Published application
- Kind: A1
- Application number: 17/204,896
- Filed: 2021-03-17
- Published: 2021-08-26
- Earliest claimed: 2018-06-01
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Karl Ferdinand SCHRAMM; Justin BINDER; Benjamin S. PHIPPS; Po Keng SUNG
- Web page: https://patentgrep.com/patent/US20210264913A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- H04M 3/42212: {Call pickup (comprising simultaneous alerting H04M3/46)}
- H04M 3/42263: {where the same subscriber uses different terminals, i.e. nomadism}
- H04M 3/527: Centralised call answering arrangements not requiring operator intervention
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- H04M 2201/40: using speech recognition
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

The present disclosure generally relates to using voice interaction to access call functionality of a companion device. In an example process, a user utterance is received. Based on the user utterance and contextual information, the process causes a server to determine a user intent corresponding to the user utterance. The contextual information is based on a signal received from the companion device. In accordance with the user intent corresponding to an actionable intent of answering the incoming call, a command is received. Based on the command, instructions are provided to the companion device, which cause the companion device to answer the incoming call and provide audio data of the answered incoming call. Audio is outputted according to the audio data of the answered incoming call.

## Claims

### Claim 1

(canceled)

### Claim 2

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a speaker, cause the electronic device to:
receive audio data of a communication session, wherein the audio data represents an utterance received from a far-end party of the active call via a communication session connection;
determine, based on the audio data, a user intent corresponding to the utterance;
determine one or more tasks to satisfy the user intent;
determine, based on results of performing the one or more tasks, a digital assistant response; and
transmit the digital assistant response to the far-end party via the communication session connection.

### Claim 3 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the far-end party is a second electronic device having a processor, memory, and a speaker, and
wherein transmitting the digital assistant response to the far-end party via the communication session connection causes the speaker of the far-end party to output the digital assistant response as a first audio signal during the communication session.

### Claim 4 (depends on 3)

The non-transitory computer-readable storage medium of claim 3, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the electronic device to:
after transmitting the digital assistant response to the far-end party, output the digital assistant response as a second audio signal via the speaker of the electronic device.

### Claim 5 (depends on 4)

The non-transitory computer-readable storage medium of claim 4, wherein the far-end party and the electronic device concurrently output the first audio signal and the second audio signal, respectively.

### Claim 6 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein determining the user intent corresponding to the utterance is based on results of performing natural language processing of the audio data.

### Claim 7 (depends on 6)

The non-transitory computer-readable storage medium of claim 6, wherein determining the user intent corresponding to the utterance is further based on contextual information.

### Claim 8 (depends on 7)

The non-transitory computer-readable storage medium of claim 7, wherein the contextual information indicates that the electronic device is engaged in the communication session.

### Claim 9 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the electronic device to:
determine whether the user intent corresponds to an actionable intent for controlling call functionality for the communication session; and
perform determining the digital assistant response and transmitting the digital assistant response in accordance with a determination that the user intent corresponds to the actionable intent for controlling communication session functionality for the communication session.

### Claim 10 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the electronic device to:
in response to determining the one or more tasks to satisfy the user intent, determining, based on the user intent, which electronic device of a plurality of electronic devices is to execute the command, wherein the plurality of electronic devices includes the electronic device.

### Claim 11

A method for using voice interaction to access functionality of a digital assistant, the method comprising:
at the electronic device having a processor, memory, and a speaker:
receiving audio data of a communication session, wherein the audio data represents an utterance received from a far-end party of the active call via a communication session connection;
determining, based on the audio data, a user intent corresponding to the utterance;
determining one or more tasks to satisfy the user intent;
determining, based on results of performing the one or more tasks, a digital assistant response; and
transmitting the digital assistant response to the far-end party via the communication session connection.

### Claim 12

An electronic device, comprising:
a speaker;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving audio data of a communication session, wherein the audio data represents an utterance received from a far-end party of the active call via a communication session connection;
determining, based on the audio data, a user intent corresponding to the utterance;
determining one or more tasks to satisfy the user intent;
determining, based on results of performing the one or more tasks, a digital assistant response; and
transmitting the digital assistant response to the far-end party via the communication session connection.
