# US 2025/0095634 A1 · Language Agnostic Multilingual End-To-End Streaming On-Device ASR System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095634 A1
- Document ID: US20250095634A1
- Status: Published application
- Kind: A1
- Application number: 18/965,193
- Filed: 2024-12-02
- Published: 2025-03-20
- Earliest claimed: 2021-10-06
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Bo Li; Tara N. Sainath; Ruoming Pang; Shuo-yiin Chang; Qiumin Xu; Trevor Strohman; Vince Chen; Qiao Liang; Heguang Liu; Yanzhang He; Parisa Haghani; Sameer Bidichandani
- Web page: https://patentgrep.com/patent/US20250095634A1

## CPC

- G10L 15/005: {Language recognition}
- G10L 15/16: using artificial neural networks
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 2015/226: {using non-speech characteristics}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method includes receiving a sequence of acoustic frames characterizing one or more utterances as input to a multilingual automated speech recognition (ASR) model. The method also includes generating a higher order feature representation for a corresponding acoustic frame. The method also includes generating a hidden representation based on a sequence of non-blank symbols output by a final softmax layer. The method also includes generating a probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps. The method also includes predicting an end of utterance (EOU) token at an end of each utterance. The method also includes classifying each acoustic frame as either speech, initial silence, intermediate silence, or final silence.

## Claims

### Claim 1

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving, as input to an automated speech recognition (ASR) model, a sequence of acoustic frames characterizing one or more utterances;
generating, by an encoder of the ASR model, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames, wherein the encoder comprises a stack of multi-headed attention layers;
generating, by a joint network of the ASR model, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses based on the higher order feature representation generated by the encoder at each of the plurality of output steps; and
generating, by an endpointer model, at each of the plurality of output steps, a classification for the higher order feature representation generated for each acoustic frame in the sequence of acoustic frames as either speech, initial silence, intermediate silence, or final silence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise triggering, by a microphone closer, a microphone closing event in response to the endpointer model classifying a higher order feature representation for a corresponding acoustic frame as final silence.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the endpointer branches from the encoder and shares a computation of one of the multi-headed attention layers from the stack of multi-headed attention layers.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the ASR model comprises a multilingual ASR model trained on multilingual training utterances sampled from a plurality of different languages.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein each multilingual training utterance is concatenated with a corresponding domain identification (ID).

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein:
multilingual training utterances concatenated with a corresponding domain ID representing a voice search domain comprise end of utterance (EOU) training tokens; and
multilingual training utterances concatenated with a corresponding domain ID representing a domain other than the voice search domain do not include any EOU training tokens.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the sequence of acoustic frames characterizes a first utterance spoken in a first language followed by a second utterance spoken in a second language different than the first language.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the stack of multi-headed attention layers comprises a stack of conformer layers.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein:
the higher order feature representation generated at each of the plurality of output steps comprises a first higher order feature representation; and
the encoder comprises:
a first sub-encoder configured to generate, at each of the plurality of output steps, the first higher order feature representation for the corresponding acoustic frame in the sequence of acoustic frames; and
a second sub-encoder configured to receive, as input, the first higher feature representation generated at each of the plurality of output steps and generate, at each of the plurality of output steps, a second higher order feature representation for the corresponding acoustic frame in the sequence of acoustic frames.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the operations further comprise:
generating, by a prediction network of the ASR model, at each of the plurality of output steps, a hidden representation based on a sequence of non-blank symbols output by a final softmax layer,
wherein generating the probability distribution over possible speech recognition hypotheses comprises generating, by the joint network of the ASR model, at each of the plurality of output steps, the probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the second higher order feature representation generated by the second sub-encoder at each of the plurality of output steps.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving, as input to an automated speech recognition (ASR) model, a sequence of acoustic frames characterizing one or more utterances;
generating, by an encoder of the ASR model, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames, wherein the encoder comprises a stack of multi-headed attention layers;
generating, by a joint network of the ASR model, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses based on the higher order feature representation generated by the encoder at each of the plurality of output steps; and
generating, by an endpointer model, at each of the plurality of output steps, a classification for the higher order feature representation generated for each acoustic frame in the sequence of acoustic frames as either speech, initial silence, intermediate silence, or final silence.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise triggering, by a microphone closer, a microphone closing event in response to the endpointer model classifying a higher order feature representation for a corresponding acoustic frame as final silence.

### Claim 13 (depends on 12)

The system of claim 12, wherein the endpointer branches from the encoder and shares a computation of one of the multi-headed attention layers from the stack of multi-headed attention layers.

### Claim 14 (depends on 11)

The system of claim 11, wherein the ASR model comprises a multilingual ASR model trained on multilingual training utterances sampled from a plurality of different languages.

### Claim 15 (depends on 14)

The system of claim 14, wherein each multilingual training utterance is concatenated with a corresponding domain identification (ID).

### Claim 16 (depends on 15)

The system of claim 15, wherein:
multilingual training utterances concatenated with a corresponding domain ID representing a voice search domain comprise end of utterance (EOU) training tokens; and
multilingual training utterances concatenated with a corresponding domain ID representing a domain other than the voice search domain do not include any EOU training tokens.

### Claim 17 (depends on 11)

The system of claim 11, wherein the sequence of acoustic frames characterizes a first utterance spoken in a first language followed by a second utterance spoken in a second language different than the first language.

### Claim 18 (depends on 11)

The system of claim 11, wherein the stack of multi-headed attention layers comprises a stack of conformer layers.

### Claim 19 (depends on 11)

The system of claim 11, wherein:
the higher order feature representation generated at each of the plurality of output steps comprises a first higher order feature representation; and
the encoder comprises:
a first sub-encoder configured to generate, at each of the plurality of output steps, the first higher order feature representation for the corresponding acoustic frame in the sequence of acoustic frames; and
a second sub-encoder configured to receive, as input, the first higher feature representation generated at each of the plurality of output steps and generate, at each of the plurality of output steps, a second higher order feature representation for the corresponding acoustic frame in the sequence of acoustic frames.

### Claim 20 (depends on 19)

The system of claim 19, wherein the operations further comprise:
generating, by a prediction network of the ASR model, at each of the plurality of output steps, a hidden representation based on a sequence of non-blank symbols output by a final softmax layer,
wherein generating the probability distribution over possible speech recognition hypotheses comprises generating, by the joint network of the ASR model, at each of the plurality of output steps, the probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the second higher order feature representation generated by the second sub-encoder at each of the plurality of output steps.
