# US 2023/0410797 A1 · Accuracy of streaming rnn transducer

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410797 A1
- Document ID: US20230410797A1
- Status: Published application
- Kind: A1
- Application number: 18/460,031
- Filed: 2023-09-01
- Published: 2023-12-21
- Earliest claimed: 2020-09-24
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Gakuto Kurata; George Andrei Saon
- Web page: https://patentgrep.com/patent/US20230410797A1

## CPC

- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method is provided for model training. The method includes training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder. The method also includes building a third end-to-end neural speech recognition model that has a unidirectional encoder by training the third end-to-end neural speech recognition model as a student by using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method.

## Claims

### Claim 1

A computer-implemented method for model training, comprising:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder; and
building a third end-to-end neural speech recognition model configured to stream output data and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein training the second end-to-end speech recognition model includes:
obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model; and
minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein a weighted summation of the cross entropy loss and the end-to-end speech recognition model loss is minimized.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein training the third end-to-end neural speech recognition model includes:
masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and
jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the relative divergence is a Kullback-Leibler divergence.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein training the second end-to-end speech recognition model further includes inputting training audio data into the trained first end-to-end neural speech recognition model to obtain the output probability lattice of the second end-to-end neural speech recognition model.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the first and third end-to-end neural speech recognition models stream the output data from a RNN transducer, and the second end-to-end neural speech recognition model provides the output data offline.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein each node of the output probability lattice of the second end-to-end neural speech recognition model represents a softmax operation.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising performing speech recognition by performing a beam search over an output probability lattice of the trained third end-to-end neural speech recognition model.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the second end-to-end neural speech recognition model comprises a more complex neural network than the first end-to-end neural speech recognition model.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising evaluating the output probability lattice of the third end-to-end neural speech recognition model based on a similarity to the output probability lattice of the second end-to-end neural speech recognition model.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the third end-to-end neural speech recognition model is trained until a similarity between the output probability lattices of the third and second end-to-end neural speech recognition models is within a particular similarity range.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the method is performed by a speech recognition system.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the first, second and third neural speech recognition models comprise a Recurrent Neural Network Transducer model.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein the second end-to-end neural speech recognition model comprises an ensemble of end-to-end neural speech recognition models, and the method further comprises selecting a best teacher model from the ensemble responsive to an overlap of search paths through an output probability lattice of each of the end-to-end neural speech recognition models in the ensemble.

### Claim 16

A computer program product for model training, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder; and
building a third end-to-end neural speech recognition model configured to stream output data and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein training the second end-to-end speech recognition model includes:
obtaining a 1-best symbol for each of nodes of the output probability lattice of the first end-to-end neural speech recognition model; and
minimizing an end-to-end neural speech recognition model loss and also a cross entropy loss regarding the 1-best symbol.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein a weighted summation of the cross-entropy loss and the end-to-end speech recognition model loss is minimized.

### Claim 19 (depends on 17)

The computer program product of claim 17, wherein training the third end-to-end neural speech recognition model includes:
masking less probable nodes of the output probability lattice of the second end-to-end neural speech recognition model with respect to a minimum probability threshold; and
jointly minimizing (i) the end-to-end neural speech recognition model loss and (ii) a relative divergence over unmasked portions of the output probability lattice of the second end-to-end neural speech recognition model to an output probability lattice of the third end-to-end speech recognition model.

### Claim 20

A computer processing system for model training, comprising:
a memory device for storing program code; and
a hardware processor device operatively coupled to the memory device for running the program code, the hardware processor device configured for:
training a second end-to-end neural speech recognition model that has a bidirectional encoder to output same symbols from an output probability lattice of the second end-to-end neural speech recognition model as from an output probability lattice of a trained first end-to-end neural speech recognition model having a unidirectional encoder; and
building a third end-to-end neural speech recognition model configured to stream output data and comprising a unidirectional encoder by training the third end-to-end speech recognition model as a student using the trained second end-to-end neural speech recognition model as a teacher in a knowledge distillation method, with output data being provided offline.
