# US 2025/0166614 A1 · Supervised and Unsupervised Training with Contrastive Loss Over Sequences

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166614 A1
- Document ID: US20250166614A1
- Status: Published application
- Kind: A1
- Application number: 19/034,304
- Filed: 2025-01-22
- Published: 2025-05-22
- Earliest claimed: 2021-03-26
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Andrew Rosenberg; Bhuvana Ramabhadran; Zhehuai Chen; Yuan Wang; Yu Zhang; Jesse Emond
- Web page: https://patentgrep.com/patent/US20250166614A1

## CPC

- G10L 15/063: {Training}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method includes receiving audio data corresponding to an utterance and generating a pair of positive audio data examples. Here, each positive audio data example includes a respective augmented copy of the received audio data. For each respective positive audio data example, the method includes generating a respective sequence of encoder outputs and projecting the respective sequence of encoder outputs for the positive data example into a contrastive loss space. The method also includes determining a L2 distance between each corresponding encoder output in the projected sequences of encoder outputs for the positive audio data examples and determining a per-utterance consistency loss by averaging the L2 distances. The method also includes generating corresponding speech recognition results for each respective positive audio data example. The method also includes updating parameters of the speech recognition model based on a respective supervised loss term and the per-utterance consistency loss.

## Claims

### Claim 1

A computer-implemented method executing on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving audio data corresponding to an utterance of non-synthetic speech;
generating, using a data augmentation module, a pair of augmented copies of the received audio data corresponding to the utterance, the pair of augmented copies of the received audio data comprising:
a first augmented copy of the received audio data having a first amount of augmentation applied by the data augmentation module; and
a second augmented copy of the received audio data having a second amount of augmentation applied by the data augmentation module, the second amount of augmentation greater than the first amount of data augmentation;
for each respective augmented copy in the pair of augmented copies of the received audio data:
generating, using an encoder, a respective sequence of encoder outputs;
projecting, using a convolutional neural network (CNN), the respective sequence of encoder outputs for the respective augmented copy in a contrastive loss space; and
generating, using a speech recognition model, a respective speech recognition result;
determining a per-utterance consistency loss based on the respective sequences of encoder outputs for the pair of augmented copies of the received audio data projected into the contrastive loss space; and
updating parameters of the speech recognition model based on a respective supervised loss term associated with each of the respective speech recognition results and the per-utterance consistency loss.

### Claim 2 (depends on 1)

The method of claim 1, wherein the encoder comprises a Conformer-based encoder.

### Claim 3 (depends on 1)

The method of claim 1, wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

### Claim 4 (depends on 3)

The method of claim 3, wherein the RNN-T architecture comprises:
a Conformer-based encoder;
a long short-term memory (LSTM)-based prediction network; and
a LSTM-based joint network.

### Claim 5 (depends on 4)

The method of claim 4, wherein the Conformer-based encoder comprises a stack of conformer layers each comprising a series of multi-headed self-attention, depth-wise convolution, and feedforward layers.

### Claim 6 (depends on 1)

The method of claim 1, wherein the CNN comprises a first CNN layer, followed by a rectified linear activation function (ReLU) activation and LayerNorm layer, and a second CNN layer with linear activation.

### Claim 7 (depends on 1)

The method of claim 1, wherein the data augmentation module adds at least one of noise, reverberation, or manipulates timing of the received audio data.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the respective speech recognition result for each respective augmented copy in the pair of augmented copies of the received audio data comprises determining, using a decoder, a probability distribution over possible speech recognition hypotheses for the respective sequence of encoder outputs.

### Claim 9 (depends on 1)

The method of claim 1, wherein the operations further comprise determining the respective supervised loss term by comparing the respective speech recognition result for the respective augmented copy and a corresponding ground-truth transcription of the utterance.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the pair of augmented copies of the received audio data corresponding to the utterance comprises generating each augmented copy in the pair of augmented copies based on a single observation of the utterance.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving audio data corresponding to an utterance of non-synthetic speech;
generating, using a data augmentation module, a pair of augmented copies of the received audio data corresponding to the utterance, the pair of augmented copies of the received audio data comprising:
a first augmented copy of the received audio data having a first amount of augmentation applied by the data augmentation module; and
a second augmented copy of the received audio data having a second amount of augmentation applied by the data augmentation module, the second amount of augmentation greater than the first amount of data augmentation;
for each respective augmented copy in the pair of augmented copies of the received audio data:
generating, using an encoder, a respective sequence of encoder outputs;
projecting, using a convolutional neural network (CNN), the respective sequence of encoder outputs for the respective augmented copy in a contrastive loss space; and
generating, using a speech recognition model, a respective speech recognition result;
determining a per-utterance consistency loss based on the respective sequences of encoder outputs for the pair of augmented copies of the received audio data projected into the contrastive loss space; and
updating parameters of the speech recognition model based on a respective supervised loss term associated with each of the respective speech recognition results and the per-utterance consistency loss.

### Claim 12 (depends on 11)

The system of claim 11, wherein the encoder comprises a Conformer-based encoder.

### Claim 13 (depends on 11)

The system of claim 11, wherein the speech recognition model comprises a recurrent neural network-transducer (RNN-T) architecture.

### Claim 14 (depends on 13)

The system of claim 13, wherein the RNN-T architecture comprises:
a Conformer-based encoder;
a long short-term memory (LSTM)-based prediction network; and
a LSTM-based joint network.

### Claim 15 (depends on 14)

The system of claim 14, wherein the Conformer-based encoder comprises a stack of conformer layers each comprising a series of multi-headed self-attention, depth-wise convolution, and feedforward layers.

### Claim 16 (depends on 11)

The system of claim 11, wherein the CNN comprises a first CNN layer, followed by a rectified linear activation function (ReLU) activation and LayerNorm layer, and a second CNN layer with linear activation.

### Claim 17 (depends on 11)

The system of claim 11, wherein the data augmentation module adds at least one of noise, reverberation, or manipulates timing of the received audio data.

### Claim 18 (depends on 11)

The system of claim 11, wherein generating the respective speech recognition result for each respective augmented copy in the pair of augmented copies of the received audio data comprises determining, using a decoder, a probability distribution over possible speech recognition hypotheses for the respective sequence of encoder outputs.

### Claim 19 (depends on 11)

The system of claim 11, wherein the operations further comprise determining the respective supervised loss term by comparing the respective speech recognition result for the respective augmented copy and a corresponding ground-truth transcription of the utterance.

### Claim 20 (depends on 11)

The system of claim 11, wherein generating the pair of augmented copies of the received audio data corresponding to the utterance comprises generating each augmented copy in the pair of augmented copies based on a single observation of the utterance.
