# US 2024/0203399 A1 · Training A Language Model Of An End-To-End Automatic Speech Recognition Model Using Random Encoder Features

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203399 A1
- Document ID: US20240203399A1
- Status: Published application
- Kind: A1
- Application number: 18/065,692
- Filed: 2022-12-14
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Adam Michael Stooke; Khe Chai Sim; Mason Vijay Chua
- Web page: https://patentgrep.com/patent/US20240203399A1

## CPC

- G10L 15/063: {Training}
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/09: Supervised learning
- G10L 15/065: Adaptation
- G10L 15/16: using artificial neural networks
- G10L 15/183: using context dependencies, e.g. language models
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method includes obtaining a training text sample, the training text sample not paired with corresponding audio data, and generating a sequence of pseudo-random encoder variables. The method also includes processing, using a decoder of a sequence transduction model, the sequence of pseudo-random encoder variables to predict a probability distribution over possible output labels. The method further includes determining a loss based metric based on the training text sample and the predicted probability distribution over possible output labels, and training the decoder based on the loss metric.

## Claims

### Claim 1

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a training text sample, the training text sample not paired with corresponding audio data;
generating a sequence of pseudo-random encoder variables;
processing, using a decoder of a sequence transduction model, the sequence of pseudo-random encoder variables to predict a probability distribution over possible output labels;
determining a loss metric based on the training text sample and the predicted probability distribution over possible output labels; and
training the decoder based on the loss metric.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein training the decoder based on the loss metric comprises training the decoder based on a negative log of the probability distribution over possible output labels for the training text sample conditioned on the sequence of pseudo-random encoder variables.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein:
the sequence transduction model is pre-trained on paired audio-text samples; and
training the decoder based on the loss metric comprises fine-tuning the decoder of the sequence transduction model based on the loss metric.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein a number of sequence pseudo-random encoder variables generated increases as a number of text labels in the training text sample increases.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise generating, by a prediction network, a corresponding text label representation for each prior sequence of non-blank symbols output by a final softmax layer of the sequence transduction model.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein processing the sequence of pseudo-random encoder variables comprises processing the sequence of pseudo-random encoder variables and the corresponding text label representations to predict the probability distribution over the possible output labels.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein the decoder comprises the prediction network and a joint network, the joint network configured to receive each corresponding text label representation generated by the prediction network and each pseudo-random encoder variable in the sequence of pseudo-random encoder variables.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein training the decoder comprises:
updating coefficients of the joint network;
holding coefficients of the prediction network fixed; and
holding coefficients of an encoder network of the sequence transduction model fixed.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the prediction network comprises a multi-headed attention mechanism, the multi-headed attention mechanism sharing a shared embedding matrix across each head of the multi-headed attention mechanism.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the prediction network ties a dimensionality of the shared embedding matrix to a dimensionality of an output layer of the joint network.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the sequence transduction model comprises a recurrent neural network-transducer (RNN-T) based speech recognition model.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein training the decoder comprises training the decoder with hybrid autoregressive transducer factorization to integrate a language model of the sequence transduction model.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the language model comprises a neural network model comprising a stack of conformer layers or transformer layers.

### Claim 14

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a training text sample, the training text sample not paired with corresponding audio data;
generating a sequence of pseudo-random encoder variables;
processing, using a decoder of a sequence transduction model, the sequence of pseudo-random encoder variables to predict a probability distribution over possible output labels;
determining a loss metric based on the training text sample and the predicted probability distribution over possible output labels; and
training the decoder based on the loss metric.

### Claim 15 (depends on 14)

The system of claim 14, wherein training the decoder based on the loss metric comprises training the decoder based on a negative log of the probability distribution over possible output labels for the training text sample conditioned on the sequence of pseudo-random encoder variables.

### Claim 16 (depends on 14)

The system of claim 14, wherein:
the sequence transduction model is pre-trained on paired audio-text samples; and
training the decoder based on the loss metric comprises fine-tuning the decoder of the sequence transduction model based on the loss metric.

### Claim 17 (depends on 14)

The system of claim 14, wherein a number of sequence pseudo-random encoder variables generated increases as a number of text labels in the training text sample increases.

### Claim 18 (depends on 14)

The system of claim 14, wherein the operations further comprise generating, by a prediction network, a corresponding text label representation for each prior sequence of non-blank symbols output by a final softmax layer of the sequence transduction model.

### Claim 19 (depends on 18)

The system of claim 18, wherein processing the sequence of pseudo-random encoder variables comprises processing the sequence of pseudo-random encoder variables and the corresponding text label representations to predict the probability distribution over the possible output labels.

### Claim 20 (depends on 18)

The system of claim 18, wherein the decoder comprises the prediction network and a joint network, the joint network configured to receive each corresponding text label representation generated by the prediction network and each pseudo-random encoder variable in the sequence of pseudo-random encoder variables.

### Claim 21 (depends on 20)

The system of claim 20, wherein training the decoder comprises:
updating coefficients of the joint network;
holding coefficients of the prediction network fixed; and
holding coefficients of an encoder network of the sequence transduction model fixed.

### Claim 22 (depends on 20)

The system of claim 20, wherein the prediction network comprises a multi-headed attention mechanism, the multi-headed attention mechanism sharing a shared embedding matrix across each head of the multi-headed attention mechanism.

### Claim 23 (depends on 22)

The system of claim 22, wherein the prediction network ties a dimensionality of the shared embedding matrix to a dimensionality of an output layer of the joint network.

### Claim 24 (depends on 14)

The system of claim 14, wherein the sequence transduction model comprises a recurrent neural network-transducer (RNN-T) based speech recognition model.

### Claim 25 (depends on 14)

The system of claim 14, wherein training the decoder comprises training the decoder with hybrid autoregressive transducer factorization to integrate a language model of the sequence transduction model.

### Claim 26 (depends on 25)

The system of claim 25, wherein the language model comprises a neural network model comprising a stack of conformer layers or transformer layers.
