# US 12,014,729 B2 · Mixture model attention for flexible streaming and non-streaming automatic speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,014,729 B2
- Document ID: US12014729B2
- Status: Granted
- Kind: B2
- Application number: 17/644,344
- Filed: 2021-12-15
- Published: 2022-09-29
- Granted: 2024-06-18
- Earliest claimed: 2021-03-26
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Kartik Audhkhasi; Bhuvana Ramabhadran; Tongzhou Chen; Pedro J. Moreno Mengibar
- Web page: https://patentgrep.com/patent/US12014729B2

## CPC

- G10L 15/16: using artificial neural networks
- G06F 1/03: working, at least partly, by table look-up (G06F1/025 takes precedence)
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 19/167: {Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes}
- G10L 15/063: {Training}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for an automated speech recognition (ASR) model for unifying streaming and non-streaming speech recognition including receiving a sequence of acoustic frames. The method includes generating, using an audio encoder of an automatic speech recognition (ASR) model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The method further includes generating, using a joint encoder of the ASR model, a probability distribution over possible speech recognition hypothesis at the corresponding time step based on the higher order feature representation generated by the audio encoder at the corresponding time step. The audio encoder comprises a neural network that applies mixture model (MiMo) attention to compute an attention probability distribution function (PDF) using a set of mixture components of softmaxes over a context window.

## Claims

### Claim 1

Data processing hardware executing instructions stored on memory hardware that causes the data processing hardware to execute an automated speech recognition (ASR) model for unifying streaming and non-streaming speech recognition, the ASR model comprising:
an audio encoder configured to:
receive, as input, a sequence of acoustic frames; and
generate, at each of a plurality of time steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and
a joint network configured to:
receive, as input, the higher order feature representation generated by the audio encoder at each of the plurality of time steps; and
generate, at each of the plurality of time steps, a probability distribution over possible speech recognition hypothesis at the corresponding time step,
wherein the audio encoder comprises a neural network that applies mixture model (MiMo) attention to compute an attention probability distribution function (PDF) using a set of mixture components of softmaxes over a context window, and
wherein the ASR model switches between streaming and non-streaming modes by adjusting mixture weights of the MiMo attention.

### Claim 2 (depends on 1)

The data processing hardware of claim 1, wherein the set of mixture components of the MiMo attention are designed to cover all possible context use cases during inference.

### Claim 3 (depends on 1)

The data processing hardware of claim 1, wherein each mixture component of the set of mixture components comprises a fully-normalized softmax.

### Claim 4 (depends on 1)

The data processing hardware of claim 1, wherein the neural network of the audio encoder comprises a plurality of conformer layers.

### Claim 5 (depends on 1)

The data processing hardware of claim 1, wherein the neural network of the audio encoder comprises a plurality of transformer layers.

### Claim 6 (depends on 1)

The data processing hardware of claim 1, further comprising a label encoder configured to:
receive, as input, a sequence of non-blank symbols output by a final softmax layer; and
generate, at each of the plurality of time steps, a dense representation,
wherein the joint network is further configured to receive, as input, the dense representation generated by the label encoder at each of the plurality of time steps.

### Claim 7 (depends on 6)

The data processing hardware of claim 6, wherein the label encoder comprises a neural network of transformer layers, conformer layers, or long short-term memory (LSTM) layers.

### Claim 8 (depends on 6)

The data processing hardware of claim 6, wherein the label encoder comprises a look-up table embedding model configured to look-up the dense representation at each of the plurality of time steps.

### Claim 9 (depends on 1)

The data processing hardware of claim 1, wherein possible speech recognition hypotheses generated at each of the plurality of time steps corresponds to a set of output labels each representing a grapheme or a word piece in a natural language.

### Claim 10

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames;
at each of a plurality of time steps:
generating, using an audio encoder of an automatic speech recognition (ASR) model, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames; and
generating, using a joint encoder of the ASR model, a probability distribution over possible speech recognition hypothesis at the corresponding time step based on the higher order feature representation generated by the audio encoder at the corresponding time step,
wherein the audio encoder comprises a neural network that applies mixture model (MiMo) attention to compute an attention probability distribution function (PDF) using a set of mixture components of softmaxes over a context window, and
wherein the ASR model switches between streaming and non-streaming modes by adjusting mixture weights of the MiMo attention.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the set of mixture components of the MiMo attention are designed to cover all possible context use cases during inference.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein each mixture component of the set of mixture components comprises a fully-normalized softmax.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein the neural network of the audio encoder comprises a plurality of conformer layers.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, wherein the neural network of the audio encoder comprises a plurality of transformer layers.

### Claim 15 (depends on 10)

The computer-implemented method of claim 10, wherein the operations further comprise:
generating, at each of the plurality of time steps, using a label encoder of the ASR model configured to receive a sequence of non-blank symbols output by a final softmax layer of the ASR model, a dense representation,
wherein generating the probability distribution over possible speech recognition hypothesis at the corresponding time step is further based on the sequence of non-blank output symbols output by the final softmax layer at the corresponding time step.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein the label encoder comprises a neural network of transformer layers, conformer layers, or long short-term memory (LSTM) layers.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15, wherein the label encoder comprises a look-up table embedding model configured to look-up the dense representation at each of the plurality of time steps.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, wherein possible speech recognition hypotheses generated at each of the plurality of time steps corresponds to a set of output labels each representing a grapheme or a word piece in a natural language.