# US 12,183,322 B2 · Language agnostic multilingual end-to-end streaming on-device ASR system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,183,322 B2
- Document ID: US12183322B2
- Status: Granted
- Kind: B2
- Application number: 17/934,555
- Filed: 2022-09-22
- Published: 2023-04-06
- Granted: 2024-12-31
- Earliest claimed: 2021-10-06
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Bo Li; Tara N. Sainath; Ruoming Pang; Shuo-yiin Chang; Qiumin Xu; Trevor Strohman; Vince Chen; Qiao Liang; Heguang Liu; Yanzhang He; Parisa Haghani; Sameer Bidichandani
- Web page: https://patentgrep.com/patent/US12183322B2

## CPC

- G10L 15/005: {Language recognition}
- G10L 15/16: using artificial neural networks
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/226: {using non-speech characteristics}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method includes receiving a sequence of acoustic frames characterizing one or more utterances as input to a multilingual automated speech recognition (ASR) model. The method also includes generating a higher order feature representation for a corresponding acoustic frame. The method also includes generating a hidden representation based on a sequence of non-blank symbols output by a final softmax layer. The method also includes generating a probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps. The method also includes predicting an end of utterance (EOU) token at an end of each utterance. The method also includes classifying each acoustic frame as either speech, initial silence, intermediate silence, or final silence.

## Claims

### Claim 1

A multilingual automated speech recognition (ASR) system comprising:
a multilingual ASR model comprising:
an encoder comprising a stack of multi-headed attention layers, the encoder configured to:
receive, as input, a sequence of acoustic frames characterizing one or more utterances; and
generate, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;
a prediction network configured to
receive, as input, a sequence of non-blank symbols output by a final softmax layer; and
generate, at each of the plurality of output steps, a hidden representation;
a first joint network configured to:
receive, as input, the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps; and
generate, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses;
a second joint network configured to:
receive, as input, the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps; and
predict, as output, an end of utterance (EOU) token at an end of each utterance; and
a multilingual endpointer model configured to classify each acoustic frame in the sequence of acoustic frames as either speech, initial silence, intermediate silence, or final silence.

### Claim 2 (depends on 1)

The multilingual ASR system of claim 1, further comprising a microphone closer configured to trigger a microphone closing event in response to each EOU token predicted as output from the second joint network.

### Claim 3 (depends on 1)

The multilingual ASR system of claim 1, further comprising a microphone closer configured to trigger a microphone closing event in response to the multilingual endpointer model classifying an acoustic frame as final silence.

### Claim 4 (depends on 3)

The multilingual ASR system of claim 3, wherein the multilingual endpointer model branches from the encoder and shares a computation of one of the multi-headed attention layers from the stack of multi-headed attention layers.

### Claim 5 (depends on 1)

The multilingual ASR system of claim 1, wherein:
the multilingual ASR model is trained using a two-stage training process;
during a first stage of the two-stage training process, the encoder, the prediction network, and the first joint network are optimized for speech recognition quality; and
during a second stage of the two-stage training process, the second joint network is initialized with the first joint network and parameters of the encoder, the prediction network, and the first joint network are frozen.

### Claim 6 (depends on 1)

The multilingual ASR system of claim 1, wherein the multilingual ASR model is trained on multilingual training utterances sampled from a plurality of different languages.

### Claim 7 (depends on 6)

The multilingual ASR system of claim 6, wherein each multilingual training utterance is concatenated with a corresponding domain identification (ID).

### Claim 8 (depends on 7)

The multilingual ASR system of claim 7, wherein:
multilingual training utterances concatenated with a corresponding domain ID representing a voice search domain comprise EOU training tokens; and
multilingual training utterances concatenated with a corresponding domain ID representing a domain other than the voice search domain do not include any EOU training tokens.

### Claim 9 (depends on 1)

The multilingual ASR system of claim 1, wherein the sequence of acoustic frames characterizes a first utterance spoken in a first language followed by a second utterance spoken in a second language different than the first language.

### Claim 10 (depends on 1)

The multilingual ASR system of claim 1, wherein the stack of multi-headed attention layers comprises a stack of conformer layers.

### Claim 11

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving, as input to a multilingual automated speech recognition (ASR) model, a sequence of acoustic frames characterizing one or more utterances;
generating, by an encoder of the multilingual ASR model, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames, wherein the encoder comprises a stack of multi-headed attention layers;
generating, by a prediction network of the multilingual ASR model, at each of the plurality of output steps, a hidden representation based on a sequence of non-blank symbols output by a final softmax layer;
generating, by a first joint network of the multilingual ASR model, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps;
predicting, by a second joint network of the multilingual ASR model, an end of utterance (EOU) token at an end of each utterance based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps; and
classifying, by a multilingual endpointer model, each acoustic frame in the sequence of acoustic frames as either speech, initial silence, intermediate silence, or final silence.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the operations further comprise triggering, by a microphone closer, a microphone closing event in response to each EOU token predicted as output from the second joint network.

### Claim 13 (depends on 11)

The computer-implemented method of claim 11, wherein the operations further comprise triggering, by a microphone closer, a microphone closing event in response to the multilingual endpointer model classifying an acoustic frame as final silence.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13, wherein the multilingual endpointer branches from the encoder and shares a computation of one of the multi-headed attention layers from the stack of multi-headed attention layers.

### Claim 15 (depends on 11)

The computer-implemented method of claim 11, wherein:
the multilingual ASR model is trained using a two-stage training process;
during a first stage of the two-stage training process, the encoder, the prediction network, and the first joint network are optimized for speech recognition quality; and
during a second stage of the two-stage training process, the second joint network is initialized with the first joint network and parameters of the encoder, the prediction network, and the first joint network are frozen.

### Claim 16 (depends on 11)

The computer-implemented method of claim 11, wherein the multilingual ASR model is trained on multilingual training utterances sampled from a plurality of different languages.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein each multilingual training utterance is concatenated with a corresponding domain identification (ID).

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein:
multilingual training utterances concatenated with a corresponding domain ID representing a voice search domain comprise EOU training tokens; and
multilingual training utterances concatenated with a corresponding domain ID representing a domain other than the voice search domain do not include any EOU training tokens.

### Claim 19 (depends on 11)

The computer-implemented method of claim 11, wherein the sequence of acoustic frames characterizes a first utterance spoken in a first language followed by a second utterance spoken in a second language different than the first language.

### Claim 20 (depends on 11)

The computer-implemented method of claim 11, wherein the stack of multi-headed attention layers comprises a stack of conformer layers.