# US 2021/0241775 A1 · Hybrid speech interface device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241775 A1
- Document ID: US20210241775A1
- Status: Published application
- Kind: A1
- Application number: 17/234,111
- Filed: 2021-04-19
- Published: 2021-08-05
- Earliest claimed: 2018-03-23
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Stanislaw Ignacy Pasko; Michal Papierski; Maciej Makowski; Marcin Fuszara
- Web page: https://patentgrep.com/patent/US20210241775A1

## CPC

- G10L 15/34: Adaptation of a single recogniser for parallel processing, e.g. by use of multiple processors or cloud computing
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A speech interface device is configured with “hybrid” capabilities, which allows the speech interface device to perform actions in response to user speech, even when the speech interface device is unable to communicate with a remote system over a wide area network (e.g., the Internet). A hybrid request selector of the speech interface device sends audio data representing user speech to both a remote speech processing system and a local speech processing component executing on the speech interface device, and then waits for a response from either or both components. The local speech processing component may start execution based on the audio data and subsequently suspend the execution until further instruction from the hybrid request selector. The hybrid request selector can then determine which response to use, and, depending on which response is chosen, may instruct the local speech processing component to either continue or terminate the suspended execution.

## Claims

### Claim 1

A method, comprising:
receiving, by a device, audio data;
sending, by the device, the audio data to a remote speech processing system;
performing, by a local speech processing component of the device, speech processing on the audio data to generate intent data indicating an intent associated with the audio data;
receiving directive data from the remote speech processing system;
determining to respond to the audio data with the directive data; and
at least one of suspending or terminating execution of the local speech processing component.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein:
the receiving the directive data comprises receiving first directive data from the remote speech processing system; and
the at least one of the suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component prior to the local speech processing component generating second directive data.

### Claim 3 (depends on 1)

The method as recited in claim 1, further comprising performing, by the device, an action based at least in part on the directive data from the remote speech processing system.

### Claim 4 (depends on 1)

The method as recited in claim 1, further comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is available for responding to the user speech.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein:
the receiving the directive data comprises receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component; and
the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component.

### Claim 6 (depends on 1)

The method as recited in claim 1, further comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is capable of responding to the audio data, and wherein the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on the receiving the data indicating that the remote speechlet is capable of responding to the audio data.

### Claim 7 (depends on 1)

The method as recited in claim 1, further comprising determining an amount of time to wait for the directive data from the remote speech processing system, and wherein:
the receiving the directive data comprises receiving the directive data from the remote speech processing system prior to expiration of the amount of time; and
the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on the receiving the directive data from the remote speech processing system prior to expiration of the amount of time.

### Claim 8

A device, comprising:
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:
receiving audio data;
sending the audio data to a remote speech processing system;
performing, by a local speech processing component of the device, speech processing on the audio data to generate intent data indicating an intent associated with the audio data;
receiving directive data from the remote speech processing system;
determining to respond to the audio data with the directive data; and
at least one of suspending or terminating execution of the local speech processing component.

### Claim 9 (depends on 8)

The device as recited in claim 8, wherein:
the receiving the directive data comprises receiving first directive data from the remote speech processing system; and
the at least one of the suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component prior to the local speech processing component generating second directive data.

### Claim 10 (depends on 8)

The device as recited in claim 8, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising performing an action based at least in part on the directive data from the remote speech processing system.

### Claim 11 (depends on 8)

The device as recited in claim 8, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is available for responding to the user speech.

### Claim 12 (depends on 8)

The device as recited in claim 8, wherein:
the receiving the directive data comprises receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component; and
the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component.

### Claim 13 (depends on 8)

The device as recited in claim 8, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is capable of responding to the audio data, and wherein the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on the receiving the data indicating that the remote speechlet is capable of responding to the audio data.

### Claim 14 (depends on 8)

The device as recited in claim 8, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising determining an amount of time to wait for the directive data from the remote speech processing system, and wherein:
the receiving the directive data comprises receiving the directive data from the remote speech processing system prior to expiration of the amount of time; and
the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on the receiving the directive data from the remote speech processing system prior to expiration of the amount of time.

### Claim 15

One or more computer-readable media storing computer-executable instructions that, when executed, cause one or more processors of a device to perform acts comprising:
receiving audio data;
sending the audio data to a remote speech processing system;
performing, by a local speech processing component of the device, speech processing on the audio data to generate intent data indicating an intent associated with the audio data;
receiving directive data from the remote speech processing system;
determining to respond to the audio data with the directive data; and
at least one of suspending or terminating execution of the local speech processing component.

### Claim 16 (depends on 15)

The one or more computer-readable media as recited in claim 15, wherein:
the receiving the directive data comprises receiving first directive data from the remote speech processing system; and
the at least one of the suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component prior to the local speech processing component generating second directive data.

### Claim 17 (depends on 15)

The one or more computer-readable media as recited in claim 15, further storing computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising performing an action based at least in part on the directive data from the remote speech processing system.

### Claim 18 (depends on 15)

The one or more computer-readable media as recited in claim 15, further storing computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is available for responding to the user speech.

### Claim 19 (depends on 15)

The one or more computer-readable media as recited in claim 15, wherein:
the receiving the directive data comprises receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component; and
the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on receiving the directive data from the remote system prior to receiving the intent data from the local speech processing component.

### Claim 20 (depends on 15)

The one or more computer-readable media as recited in claim 15, further storing computer-executable instructions that, when executed, cause the one or more processors to perform an act comprising receiving, from the remote speech processing system, data indicating that a remote speechlet is capable of responding to the audio data, and wherein the at least one of suspending or terminating comprises at least one of suspending or terminating execution of the local speech processing component based at least in part on the receiving the data indicating that the remote speechlet is capable of responding to the audio data.
