# US 2021/0280170 A1 · Consistency Prediction On Streaming Sequence Models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280170 A1
- Document ID: US20210280170A1
- Status: Published application
- Kind: A1
- Application number: 17/170,836
- Filed: 2021-02-08
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Zhehuai Chen; Andrew Rosenberg; Bhuvana Ramabhadran; Pedro Jose Moreno Mengibar
- Web page: https://patentgrep.com/patent/US20210280170A1

## CPC

- G10L 15/063: {Training}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 15/16: using artificial neural networks
- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for training a speech recognition model includes receiving a set of training utterance pairs each including a non-synthetic speech representation and a synthetic speech representation of a same corresponding utterance. At each of a plurality of output steps for each training utterance pair in the set of training utterance pairs, the method also includes determining a consistent loss term for the corresponding training utterance pair based on a first probability distribution over possible non-synthetic speech recognition hypotheses generated for the corresponding non-synthetic speech representation and a second probability distribution over possible synthetic speech recognition hypotheses generated for the corresponding synthetic speech representation. The first and second probability distributions are generated for output by the speech recognition model. The method also includes updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair.

## Claims

### Claim 1

A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
receiving a set of training utterance pairs, each training utterance pair comprising:
a non-synthetic speech representation of a corresponding utterance; and
a synthetic speech representation of the corresponding utterance;
at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs:
generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance;
generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance; and
determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses; and
updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

### Claim 2 (depends on 1)

The method of claim 1, wherein the operations further comprise, at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs:
generating a non-synthetic speech loss term based on the first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground-truth transcription of the corresponding utterance; and
generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground-truth transcription of the corresponding utterance.

### Claim 3 (depends on 2)

The method of claim 2, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs independently of the non-synthetic and synthetic speech loss terms generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

### Claim 4 (depends on 1)

The method of claim 1, wherein the synthetic speech representation in each training utterance pair comprises synthesized speech representing the corresponding utterance.

### Claim 5 (depends on 1)

The method of claim 1, wherein the operations further comprise, for each training utterance pair, converting, using a text-to-speech (TTS) model, a ground truth transcription of the corresponding utterance to generate the corresponding synthetic speech representation.

### Claim 6 (depends on 1)

The method of claim 1, wherein the operations further comprise, for at least one training utterance pair, applying data augmentation to at least one of the corresponding non-synthetic speech representation or the corresponding synthetic speech representation.

### Claim 7 (depends on 6)

The method of claim 6, wherein the applied data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining the consistent loss term for the corresponding training utterance pair is based on a Kullback-Leibler divergence between the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses.

### Claim 9 (depends on 8)

The method of claim 8, wherein the operations further comprise, prior to determining the consistent loss term for the corresponding training utterance pair, when the speech recognition model comprises a frame alignment-based transducer model, defining a valid alignment distribution for a sequence of first probability distributions over possible non-synthetic speech recognition hypotheses generated at each of the output steps and a sequence of second probability distributions over possible synthetic speech recognition hypotheses generated at each of the output steps based on a probability of valid alignments of a ground-truth transcription of the corresponding utterance.

### Claim 10 (depends on 9)

The method of claim 9, wherein the valid alignment distribution is defined over each output label in a sequence of output labels representing the ground-truth transcription of the corresponding utterance.

### Claim 11 (depends on 1)

The method of claim 1, wherein the speech recognition model comprises a frame-alignment-based transducer model.

### Claim 12 (depends on 11)

The method of claim 11, wherein the frame-alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model.

### Claim 13 (depends on 1)

The method of claim 1, wherein the speech recognition model comprises an alignment-based encoder-decoder model.

### Claim 14

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a set of training utterance pairs, each training utterance pair comprising:
a non-synthetic speech representation of a corresponding utterance; and
a synthetic speech representation of the corresponding utterance;
at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs:
generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance;
generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance; and
determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses; and
updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

### Claim 15 (depends on 14)

The system of claim 14, wherein the operations further comprise, at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs:
generating a non-synthetic speech loss term based on the first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance and a ground-truth transcription of the corresponding utterance; and
generating a synthetic speech loss term based on the second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance and the ground-truth transcription of the corresponding utterance.

### Claim 16 (depends on 15)

The system of claim 15, wherein the parameters of the speech recognition model are updated based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs independently of the non-synthetic and synthetic speech loss terms generated at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.

### Claim 17 (depends on 14)

The system of claim 14, wherein the synthetic speech representation in each training utterance pair comprises synthesized speech representing the corresponding utterance.

### Claim 18 (depends on 14)

The system of claim 14, wherein the operations further comprise, for each training utterance pair, converting, using a text-to-speech (TTS) model, a ground truth transcription of the corresponding utterance to generate the corresponding synthetic speech representation.

### Claim 19 (depends on 14)

The system of claim 14, wherein the operations further comprise, for at least one training utterance pair, applying data augmentation to at least one of the corresponding non-synthetic speech representation or the corresponding synthetic speech representation.

### Claim 20 (depends on 19)

The system of claim 19, wherein the applied data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.

### Claim 21 (depends on 14)

The system of claim 14, wherein determining the consistent loss term for the corresponding training utterance pair is based on a Kullback-Leibler divergence between the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses.

### Claim 22 (depends on 21)

The system of claim 21, wherein the operations further comprise, prior to determining the consistent loss term for the corresponding training utterance pair, when the speech recognition model comprises a frame alignment-based transducer model, defining a valid alignment distribution for a sequence of first probability distributions over possible non-synthetic speech recognition hypotheses generated at each of the output steps and a sequence of second probability distributions over possible synthetic speech recognition hypotheses generated at each of the output steps based on a probability of valid alignments of a ground-truth transcription of the corresponding utterance.

### Claim 23 (depends on 22)

The system of claim 22, wherein the valid alignment distribution is defined over each output label in a sequence of output labels representing the ground-truth transcription of the corresponding utterance.

### Claim 24 (depends on 14)

The system of claim 14, wherein the speech recognition model comprises a frame-alignment-based transducer model.

### Claim 25 (depends on 24)

The system of claim 24, wherein the frame-alignment-based transducer model comprises a Recurrent Neural Network-Transducer (RNN-T) model.

### Claim 26 (depends on 14)

The system of claim 14, wherein the speech recognition model comprises an alignment-based encoder-decoder model.
