# US 12,190,869 B2 · Optimizing inference performance for conformer

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,869 B2
- Document ID: US12190869B2
- Status: Granted
- Kind: B2
- Application number: 17/936,547
- Filed: 2022-09-29
- Published: 2023-04-27
- Granted: 2025-01-07
- Earliest claimed: 2021-10-05
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Tara N. Sainath; Rami Botros; Anmol Gulati; Krzysztof Choromanski; Ruoming Pang; Trevor Strohman; Weiran Wang; Jiahui Yu
- Web page: https://patentgrep.com/patent/US12190869B2

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G10L 15/063: {Training}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/09: Supervised learning
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method includes receiving a sequence of acoustic frames as input to an automatic speech recognition (ASR) model. Here, the ASR model includes a causal encoder and a decoder. The method also includes generating, by the causal encoder, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The method also includes generating, by the decoder, a first probability distribution over possible speech recognition hypotheses. Here, the causal encoder includes a stack of causal encoder layers each including a Recurrent Neural Network (RNN) Attention-Performer module that applies linear attention.

## Claims

### Claim 1

A system comprising data processing hardware and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations that comprise implementing an automated speech recognition (ASR) model, the ASR model comprising:
a causal encoder comprising a stack of causal encoder layers, the causal encoder configured to:
receive, as input, a sequence of acoustic frames; and
generate, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and
a decoder configured to:
receive, as input, the first higher order feature representation generated by the causal encoder at each of the plurality of output steps; and
generate, at each of the plurality of output steps, a first probability distribution over possible speech recognition hypotheses,
wherein each causal encoder layer in the stack of causal encoder layers includes a Recurrent Neural Network (RNN) Attention-Performer module that applies linear attention,
wherein, during pre-training of the ASR model, each causal encoder layer comprises:
a first feedforward module;
a convolution module;
a multi-head attention module;
a second feedforward module; and
a layernorm module.

### Claim 2 (depends on 1)

The system of claim 1, wherein, during fine-tune training of the ASR model, each causal encoder layer comprises:
the first feedforward module;
the convolution module;
the RNN Attention-Performer module;
the second feedforward module; and
the layernorm module.

### Claim 3 (depends on 2)

The system of claim 2, wherein each causal encoder layer is pre-trained using regular conformer training for the multi-head attention module and is fine-tuned by replacing the multi-head attention module with the RNN Attention-Performer module.

### Claim 4 (depends on 3)

The system of claim 3, wherein replacing the multi-head attention module with the RNN Attention-Performer module comprises applying a trainable affine transformation to convert queries/keys from the pre-trained multi-head attention module into an RNN model of linear time and constant memory complexity in sequence length.

### Claim 5 (depends on 1)

The system of claim 1, wherein the causal encoder further comprises an initial stack of convolution blocks without self-attention.

### Claim 6 (depends on 1)

The system of claim 1, wherein the ASR model further comprises:
a non-causal encoder configured to:
receive, as input, the first higher order feature representation generated by the causal encoder at each of the plurality of output steps; and
generate, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation,
wherein the decoder is further configured to:
receive, as input, the second higher order feature representation generated by the non-causal encoder at each of the plurality of output steps; and
generate, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses.

### Claim 7 (depends on 6)

The system claim 6, wherein the decoder comprises:
a prediction network configured to:
receive, as input, a sequence of non-blank symbols output by a final softmax layer; and
generate, at each of the plurality of output steps, a dense representation; and
a joint network configured to:
receive, as input, the dense representation generated by the prediction network at each of the plurality of output steps and one of:
when the ASR model is operating in a streaming mode, the first higher order feature representation generated by the causal encoder at each of the plurality of output steps; or
when the ASR model is operating in a non-streaming mode, the second higher order feature representation generated by the non-causal encoder at each of the plurality of output steps; and
generate, at each of the plurality of output steps, one of:
when the ASR model is operating in the streaming mode, the first probability distribution over possible speech recognition hypotheses; or
when the ASR model is operating in the non-streaming mode, the second probability distribution over possible speech recognition hypotheses.

### Claim 8 (depends on 7)

The system of claim 7, wherein the prediction network comprises a long short-term memory (LSTM)-based prediction network.

### Claim 9 (depends on 7)

The system of claim 7, wherein the prediction network comprises a V2 embedding look-up table.

### Claim 10

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving, as input to an automatic speech recognition (ASR) model, a sequence of acoustic frames, wherein the ASR model comprises a causal encoder and a decoder;
generating, by the causal encoder, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and
generating, by the decoder, at each of the plurality of output steps, a first probability distribution over possible speech recognition hypotheses,
wherein the causal encoder comprises a stack of causal encoder layers, each causal encoder layer in the stack of causal encoder layers comprising a Recurrent Neural Network (RNN) Attention-Performer module that applies linear attention,
wherein, during pre-training of the ASR model, each causal encoder layer comprises:
a first feedforward module;
a convolution module;
a multi-head attention module;
a second feedforward module; and
a layernorm module.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein, during fine-tune training of the ASR model, each causal encoder layer comprises:
the first feedforward module;
the convolution module;
the RNN Attention-Performer module;
the second feedforward module; and
the layernorm module.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the operations further comprise:
pre-training each causal encoder layer using regular conformer training for the multi-head attention module; and
fine-tuning each causal encoder layer by replacing the multi-head attention module with the RNN Attention-Performer module.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein replacing the multi-head attention module with the RNN Attention-Performer module comprises applying a trainable affine transformation to convert queries/keys from the pre-trained multi-head attention module into an RNN model of linear time and constant memory complexity in sequence length.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, wherein the causal encoder further comprises an initial stack of convolution blocks without self-attention.

### Claim 15 (depends on 10)

The computer-implemented method of claim 10, wherein the operations further comprise:
generating, by a non-causal encoder of the ASR model, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature representation generated by the causal encoder; and
generating, by the decoder, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses for a corresponding second higher order feature representation.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein the operations further comprise:
receiving, as input at a prediction network of the decoder, a sequence of non-blank symbols output by a final softmax layer;
generating, by the prediction network, a dense representation; and
generating, by a joint network of the decoder, one of:
when the ASR model is operating in a streaming mode, the first probability distribution over possible speech recognition hypotheses; or
when the ASR model is operating in a non-streaming mode, the second probability distribution over possible speech recognition hypotheses.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein the prediction network comprises a long short-term memory (LSTM)-based prediction network.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16, wherein the prediction network comprises a V2 embedding look-up table.