# US 2021/0233512 A1 · Proper Noun Recognition in End-to-End Speech Recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233512 A1
- Document ID: US20210233512A1
- Status: Published application
- Kind: A1
- Application number: 17/150,491
- Filed: 2021-01-15
- Published: 2021-07-29
- Earliest claimed: 2020-01-28
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Charles Caleb Peyser; Tara N. Sainath; Golan Pundak
- Web page: https://patentgrep.com/patent/US20210233512A1

## CPC

- G10L 15/063: {Training}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for training a speech recognition model with a minimum word error rate loss function includes receiving a training example comprising a proper noun and generating a plurality of hypotheses corresponding to the training example. Each hypothesis of the plurality of hypotheses represents the proper noun and includes a corresponding probability that indicates a likelihood that the hypothesis represents the proper noun. The method also includes determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criteria. The penalty criteria indicating that the corresponding probability satisfies a probability threshold, and the associated hypothesis incorrectly represents the proper noun. The method also includes applying a penalty to the minimum word error rate loss function.

## Claims

### Claim 1

A computer-implemented method that when executed on data processing hardware causes the data processing to perform operations comprising:
training a speech recognition model with a minimum word error rate loss function by:
receiving a training example comprising a proper noun;
generating a plurality of hypotheses corresponding to the training example, each hypothesis of the plurality of hypotheses representing the proper noun and comprising a corresponding probability that indicates a likelihood that the hypothesis represents the proper noun;
determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criteria, the penalty criteria indicating that:
the corresponding probability satisfies a probability threshold; and
the associated hypothesis incorrectly represents the proper noun; and
applying a penalty to the minimum word error rate loss function.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the corresponding probability satisfies the probability threshold when the corresponding probability is greater than the corresponding probabilities associated with the other hypothesis.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the speech recognition model comprises a two-pass architecture comprising:
a first pass network comprising a recurrent neural network transducer (RNN-T) decoder; and
a second pass network comprising a listen-attend-spell (LAS) decoder.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the speech recognition model further comprises a shared encoder, the shared encoder encoding acoustic frames for each of the first pass network and the second pass network.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein training with the minimum word error rate loss function occurs at the LAS decoder.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein the operations further comprise:
training the RNN-T decoder; and
prior to training the LAS decoder with the minimum word error rate loss function, training the LAS decoder while parameters of the trained RNN-T decoder remain fixed.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise assigning the probability to each hypothesis of the plurality of hypotheses.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
receiving an incorrect hypothesis; and
assigning a respective probability to the incorrect hypothesis,
wherein the penalty criteria further comprises an indication that the hypothesis includes the generated incorrect hypothesis.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the incorrect hypothesis comprises a phonetically similarity to the proper noun.

### Claim 10 (depends on 8)

The computer-implemented method of claim 8, wherein the operations further comprise substituting the incorrect hypothesis for a generated hypothesis of the plurality of hypotheses.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
training a speech recognition model with a minimum word error rate loss function by:
receiving a training example comprising a proper noun;
generating a plurality of hypotheses corresponding to the training example, each hypothesis of the plurality of hypotheses representing the proper noun and comprising a corresponding probability that indicates a likelihood that the hypothesis represents the proper noun;
determining that the corresponding probability associated with one of the plurality of hypotheses satisfies a penalty criteria, the penalty criteria indicating that:
the corresponding probability satisfies a probability threshold; and
the associated hypothesis incorrectly represents the proper noun; and
applying a penalty to the minimum word error rate loss function.

### Claim 12 (depends on 11)

The system of claim 11, wherein the corresponding probability satisfies the probability threshold when the corresponding probability is greater than the corresponding probabilities associated with the other hypothesis.

### Claim 13 (depends on 11)

The system of claim 11, further comprising:
a first pass network comprising a recurrent neural network transducer (RNN-T) decoder; and
a second pass network comprising a listen-attend-spell (LAS) decoder,
wherein the speech recognition model comprises the first pass network and the second pass network.

### Claim 14 (depends on 13)

The system of claim 13, further comprising a shared encoder configured to encode acoustic frames for each of the first pass network and the second pass network.

### Claim 15 (depends on 13)

The system of claim 13, wherein training with the minimum word error rate loss function occurs at the LAS decoder.

### Claim 16 (depends on 13)

The system of claim 13, wherein the operations further comprise:
training the RNN-T decoder; and
prior to training the LAS decoder with the minimum word error rate loss function, training the LAS decoder while parameters of the trained RNN-T decoder remain fixed.

### Claim 17 (depends on 11)

The system of claim 11, wherein the operations further comprise assigning the probability to each hypothesis of the plurality of hypotheses.

### Claim 18 (depends on 11)

The system of claim 11, wherein the operations further comprise:
receiving an incorrect hypothesis; and
assigning a respective probability to the incorrect hypothesis,
wherein the penalty criteria further comprises an indication that the hypothesis includes the generated incorrect hypothesis.

### Claim 19 (depends on 18)

The system of claim 18, wherein the incorrect hypothesis comprises a phonetically similarity to the proper noun.

### Claim 20 (depends on 18)

The system of claim 18, wherein the operations further comprise substituting the incorrect hypothesis for a generated hypothesis of the plurality of hypotheses.
