US 11,900,921 B1Granted

Multi-device speech processing

AssigneeAmazon Technologies, Inc.

InventorsRahul Gupta, Christophe Dupuy, Jacob Ryan Stolee, Clement Chung

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for partially processing an input on a device and completing processing at a remote system are provided. The device may process an input using an on-device machine learning (ML) model, and determine to cease processing at an intermediary node of the (ML) model based on the output of the intermediary node. Based on the output of the intermediary node satisfying a condition, the device may use the output of the intermediary node to generate an output responsive to the input. Conversely, if the output of the intermediary node does not satisfy a condition, the device may send the output of the intermediary node to the remote system, so the remote system can use another machine learning model to complete processing with respect to the input.

Claims20

Dependent claims

A system comprising:

a first component including at least a first processor and at least a first memory, wherein the first component is a user device;
and
a second component including at least a second processor and at least a second memory, wherein the second component is a second device that uses a network connection to communicate with the first component,
the at least first memory including first instructions that, when executed by the at least first processor, cause the first component to:
receive first audio data representing a first spoken user input;
using a first node of a first machine learning (ML) model, the first ML model configured to perform spoken language understanding (SLU) processing, process a first portion of the first audio data to determine embedding data representing a partial speech recognition output of the first node;
determine, based on the embedding data, the second component is to process to generate a SLU output;
and
in response to determining the second component is to process to generate the SLU output, and prior to processing the embedding data using a second node of the first ML model, send the embedding data to the second component,
the at least second memory including second instructions that, when executed by the at least second processor, cause the second component to:
process the embedding data using a second ML model to complete the SLU processing of the first audio data;
determine, using the second ML model, a first SLU output;
determine, using the first SLU output, a first output responsive to the first spoken user input;
and
send the first output to the first component.

12 more elements

A computer-implemented method comprising:

receiving, at a first device, first input data corresponding to a first input;
processing, at the first device, a first portion of the first input data using a second portion of a first machine learning (ML) model, wherein the second portion of the first ML model comprises a first node of the first ML model;
determining, at the first device and based at least in part on processing the first portion of the first input data using the second portion of the first ML model, first embedding data representing a partial speech recognition output of the second portion of the first ML model;
determining, based on the first embedding data, a second device is to complete the processing of the first input data;
and
in response to determining the second device is to complete the processing of the first input data, and prior to processing the first embedding data using a third portion of the first ML model, sending the first embedding data to the second device to complete the processing of the first input data using a second ML model, wherein:
the third portion of the first ML model comprises a second node of the first ML model,
the first ML model and the second ML model are configured to perform a first type of processing, and
sending the first embedding data to the second device causes the second device to complete the first type of processing of the first input data using the second ML model to determine a response to the first input.

6 more elements

A system comprising:

a first device including at least one processor, and at least one memory including instructions that, when executed by the at least one processor, cause the first device to:
receive first input data corresponding to a first input;
process a first portion of the first input data using a second portion of a first machine learning (ML) model, wherein the second portion of the first ML model comprises a first node of the first ML model;
determine, based at least in part on processing the first portion of the first input data using the second portion of the first ML model, first embedding data representing a partial speech recognition output of the second portion of the first ML model;
determine, based at least in part on the first embedding data, a second device is to complete the processing of the first input data;
and
in response to determining the second device is to complete the processing of the first input data, and prior to processing the first embedding data using a third portion of the first ML model, send the first embedding data to the second device to complete the processing of the first input data using a second ML model, wherein:
the third portion of the first ML model comprises a second node of the first ML model,
the first ML model and the second ML model are configured to perform a first type of processing, and
sending the first embedding data to the second device causes the second device to complete the first type of processing of the first input data using the second ML model to determine a response to the first input.

7 more elements

Dependencies are inferred from the claim text.