# US 11,908,461 B2 · Deliberation model-based two-pass end-to-end speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,461 B2
- Document ID: US11908461B2
- Status: Granted
- Kind: B2
- Application number: 17/149,018
- Filed: 2021-01-14
- Published: 2021-07-22
- Granted: 2024-02-20
- Earliest claimed: 2020-01-21
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ke Hu; Tara N. Sainath; Ruoming Pang; Rohit Prakash Prabhavalkar
- Web page: https://patentgrep.com/patent/US11908461B2

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L 19/0018: {Speech coding using phonetic or linguistical decoding of the source; Reconstruction using text-to-speech synthesis}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 2015/081: {Search algorithms, e.g. Baum-Welch or Viterbi}
- G10L 2015/085: {Methods for reducing search complexity, pruning}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of performing speech recognition using a two-pass deliberation architecture includes receiving a first-pass hypothesis and an encoded acoustic frame and encoding the first-pass hypothesis at a hypothesis encoder. The first-pass hypothesis is generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame. The method also includes generating, using a first attention mechanism attending to the encoded acoustic frame, a first context vector, and generating, using a second attention mechanism attending to the encoded first-pass hypothesis, a second context vector. The method also includes decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis.

## Claims

### Claim 1

A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a first-pass hypothesis and an encoded acoustic frame, the first-pass hypothesis generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame;
encoding the first-pass hypothesis at a hypothesis encoder;
generating, using an acoustic attention mechanism attending to the encoded acoustic frame, a first context vector;
generating, using a hypothesis attention mechanism attending to the encoded first-pass hypothesis, a second context vector; and
decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein decoding the first context vector and the second context vector comprises decoding a concatenation of the first context vector and the second context vector.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein encoding the first-pass hypothesis comprises bi-directionally encoding the first-pass hypothesis at the hypothesis encoder to generate contextual information from the first-pass hypothesis.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the hypothesis encoder comprises a long short term memory (LSTM) network.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
encoding the acoustic frame at a shared encoder; and
generating the first-pass hypothesis at the RNN decoder model based on the encoded acoustic frame communicated from the shared encoder.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the operations further comprise generating an acoustic embedding at a unidirectional audio encoder based on the encoded acoustic frame communicated from the shared encoder.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the unidirectional audio encoder comprises a long short term memory (LSTM) network.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the LSTM network comprises at least two layers.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
training the RNN decoder model; and
training a deliberation decoder while parameters of the trained RNN decoder model remain fixed, the deliberation decoder comprising the hypothesis encoder, the acoustic attention mechanism, the hypothesis attention mechanism, and the context vector decoder.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the operations further comprise minimizing a word error rate during training of the RNN decoder model and the deliberation decoder model.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise jointly training the RNN decoder model and a deliberation decoder, the deliberation decoder comprising the hypothesis encoder, the acoustic attention mechanism, the hypothesis attention mechanism, and the context vector decoder.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the operations further comprise minimizing a word error rate during the joint training of the RNN decoder model and the deliberation decoder model.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the data processing hardware resides on a user device.

### Claim 14

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a first-pass hypothesis and an encoded acoustic frame, the first-pass hypothesis generated by a recurrent neural network (RNN) decoder model for the encoded acoustic frame;
encoding the first-pass hypothesis at a hypothesis encoder;
generating, using an acoustic attention mechanism attending to the encoded acoustic frame, a first context vector;
generating, using a hypothesis attention mechanism attending to the encoded first-pass hypothesis, a second context vector; and
decoding the first context vector and the second context vector at a context vector decoder to form a second-pass hypothesis.

### Claim 15 (depends on 14)

The system of claim 14, wherein decoding the first context vector and the second context vector comprises decoding a concatenation of the first context vector and the second context vector.

### Claim 16 (depends on 14)

The system of claim 14, wherein encoding the first-pass hypothesis comprises bi-directionally encoding the first-pass hypothesis at the hypothesis encoder to generate contextual information from the first-pass hypothesis.

### Claim 17 (depends on 16)

The system of claim 16, wherein the hypothesis encoder comprises a long short term memory (LSTM) network.

### Claim 18 (depends on 14)

The system of claim 14, wherein the operations further comprise:
encoding the acoustic frame at a shared encoder; and
generating the first-pass hypothesis at the RNN decoder model based on the encoded acoustic frame communicated from the shared encoder.

### Claim 19 (depends on 18)

The system of claim 18, wherein the operations further comprise generating an acoustic embedding at a unidirectional audio encoder based on the encoded acoustic frame communicated from the shared encoder.

### Claim 20 (depends on 19)

The system of claim 19, wherein the unidirectional audio encoder comprises a long short term memory (LSTM) network.

### Claim 21 (depends on 20)

The system of claim 20, wherein the LSTM network comprises at least two layers.

### Claim 22 (depends on 14)

The system of claim 14, wherein the operations further comprise:
training the RNN decoder model; and
training a deliberation decoder while parameters of the trained RNN decoder model remain fixed, the deliberation decoder comprising the hypothesis encoder, the acoustic attention mechanism, the hypothesis attention mechanism, and the context vector decoder.

### Claim 23 (depends on 22)

The system of claim 22, wherein the operations further comprise minimizing a word error rate during training of the RNN decoder model and the deliberation decoder model.

### Claim 24 (depends on 14)

The system of claim 14, wherein the operations further comprise jointly training the RNN decoder model and a deliberation decoder, the deliberation decoder comprising the hypothesis encoder, the acoustic attention mechanism, the hypothesis attention mechanism, and the context vector decoder.

### Claim 25 (depends on 24)

The system of claim 24, wherein the operations further comprise minimizing a word error rate during the joint training of the RNN decoder model and the deliberation decoder model.

### Claim 26 (depends on 14)

The system of claim 14, wherein the data processing hardware resides on a user device.