# US 2025/0149026 A1 · Extremely fast utterances for measuring unintended memorization in automatic speech recognition models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0149026 A1
- Document ID: US20250149026A1
- Status: Published application
- Kind: A1
- Application number: 18/915,058
- Filed: 2024-10-14
- Published: 2025-05-08
- Earliest claimed: 2023-10-16
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Lun Wang; Om Dipakbhai Thakkar; Rajiv Mathews
- Web page: https://patentgrep.com/patent/US20250149026A1

## CPC

- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 15/065: Adaptation
- G06F 21/6245: {Protecting personal data, e.g. for financial or medical purposes}
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 15/063: {Training}
- G06N 20/00: Machine learning
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method includes obtaining an automatic speech recognition (ASR) model pre-trained on an initial training dataset, creating a set of canary speech utterances, and speeding up each canary speech utterance in the set of canary speech utterances. The operations also include fine-tuning the ASR model on the set of sped-up canary speech utterances and measuring un-intended memorization of the fine-tuned ASR model based on speech recognition results performed by the fine-tuned ASR model on the sped-up canary speech utterances.

## Claims

### Claim 1

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining an automatic speech recognition (ASR) model pre-trained on an initial training dataset;
creating a set of canary speech utterances;
speeding up each canary speech utterance in the set of canary speech utterances;
fine-tuning the ASR model on the set of sped-up canary speech utterances; and
measuring un-intended memorization of the fine-tuned ASR model based on speech recognition results performed by the fine-tuned ASR model on the sped-up canary speech utterances.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
obtaining a set of transcribed speech utterances, each transcribed speech utterance paired with a corresponding ground-truth transcription,
wherein fine-tuning the ASR model on the set of sped-up canary speech utterances further comprises fine-tuning the ASR model on the set of transcribed speech utterances.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein a number of utterances in the set of transcribed speech utterances is less than a number of utterances in the initial training data set used to pre-train the ASR model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the initial training data set used to pre-train the ASR model comprises a set of un-transcribed speech utterances that each comprise audio-only data not paired with any corresponding transcription.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the set of un-transcribed speech utterances are multilingual.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein a number of utterances in the initial training data set is greater than a number of utterances in the set of canary speech utterances.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4, wherein the ASR model is pre-trained on the set of un-transcribed speech utterances using BERT-based Speech pre-training with random projection quantizer (BEST-RQ).

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein creating the set of canary speech utterances comprises:
generating a set of text-only utterances from a language model; and
converting, using a text-to-speech (TTS) system, each text-only utterances from the set of text-only utterances into a corresponding synthesized speech representation,
wherein the synthesized speech representation converted from the set of text-only utterances form corresponding ones of the set of canary speech utterances.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the set of text-only utterances generated from the language model comprise a sequence of randomly sampled consonants and words from the language model.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein speeding up each canary speech utterance in the set of canary speech utterances comprises speeding up each canary speech utterance to a speaking pace that is faster than a normal human speaking pace.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the speaking pace of each sped-up canary speech utterance is four times faster than the normal human speaking pace.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise applying sensitivity-bounded training is applied when fine-tuning the ASR model.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the sensitivity-bounded training comprises per-core clipping wherein gradients on each GPU/TPU core on which the ASR model executes are averaged and clipping is applied on the average gradient for each GPU/TPU core.

### Claim 14

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations that include:
obtaining an automatic speech recognition (ASR) model pre-trained on an initial training dataset;
creating a set of canary speech utterances;
speeding up each canary speech utterance in the set of canary speech utterances;
fine-tuning the ASR model on the set of sped-up canary speech utterances; and
measuring un-intended memorization of the fine-tuned ASR model based on speech recognition results performed by the fine-tuned ASR model on the sped-up canary speech utterances.

### Claim 15 (depends on 14)

The system of claim 14, wherein the operations further comprise:
obtaining a set of transcribed speech utterances, each transcribed speech utterance paired with a corresponding ground-truth transcription,
wherein fine-tuning the ASR model on the set of sped-up canary speech utterances further comprises fine-tuning the ASR model on the set of transcribed speech utterances.

### Claim 16 (depends on 15)

The system of claim 15, wherein a number of utterances in the set of transcribed speech utterances is less than a number of utterances in the initial training data set used to pre-train the ASR model.

### Claim 17 (depends on 14)

The system of claim 14, wherein the initial training data set used to pre-train the ASR model comprises a set of un-transcribed speech utterances that each comprise audio-only data not paired with any corresponding transcription.

### Claim 18 (depends on 17)

The system of claim 17, wherein the set of un-transcribed speech utterances are multilingual.

### Claim 19 (depends on 17)

The system of claim 17, wherein a number of utterances in the initial training data set is greater than a number of utterances in the set of canary speech utterances.

### Claim 20 (depends on 17)

The system of claim 17, wherein the ASR model is pre-trained on the set of un-transcribed speech utterances using BERT-based Speech pre-training with random projection quantizer (BEST-RQ).

### Claim 21 (depends on 14)

The system of claim 14, wherein creating the set of canary speech utterances comprises:
generating a set of text-only utterances from a language model; and
converting, using a text-to-speech (TTS) system, each text-only utterances from the set of text-only utterances into a corresponding synthesized speech representation,
wherein the synthesized speech representation converted from the set of text-only utterances form corresponding ones of the set of canary speech utterances.

### Claim 22 (depends on 21)

The system of claim 21, wherein the set of text-only utterances generated from the language model comprise a sequence of randomly sampled consonants and words from the language model.

### Claim 23 (depends on 14)

The system of claim 14, wherein speeding up each canary speech utterance in the set of canary speech utterances comprises speeding up each canary speech utterance to a speaking pace that is faster than a normal human speaking pace.

### Claim 24 (depends on 23)

The system of claim 23, wherein the speaking pace of each sped-up canary speech utterance is four times faster than the normal human speaking pace.

### Claim 25 (depends on 14)

The system of claim 14, wherein the operations further comprise applying sensitivity-bounded training is applied when fine-tuning the ASR model.

### Claim 26 (depends on 25)

The system of claim 25, wherein the sensitivity-bounded training comprises per-core clipping wherein gradients on each GPU/TPU core on which the ASR model executes are averaged and clipping is applied on the average gradient for each GPU/TPU core.
