# US 11,908,458 B2 · Customization of recurrent neural network transducers for speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,458 B2
- Document ID: US11908458B2
- Status: Granted
- Kind: B2
- Application number: 17/136,439
- Filed: 2020-12-29
- Published: 2022-06-30
- Granted: 2024-02-20
- Earliest claimed: 2020-12-29
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Gakuto Kurata; George Andrei Saon; Brian E. D. Kingsbury
- Web page: https://patentgrep.com/patent/US11908458B2

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 15/063: {Training}
- G10L 25/30: using neural networks
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method for customizing a recurrent neural network transducer (RNN-T) is provided. The computer implemented method includes synthesizing first domain audio data from first domain text data, and feeding the synthesized first domain audio data into a trained encoder of the recurrent neural network transducer (RNN-T) having an initial condition, wherein the encoder is updated using the synthesized first domain audio data and the first domain text data. The computer implemented method further includes synthesizing second domain audio data from second domain text data, and feeding the synthesized second domain audio data into the updated encoder of the recurrent neural network transducer (RNN-T), wherein the prediction network is updated using the synthesized second domain audio data and the second domain text data. The computer implemented method further includes restoring the updated encoder to the initial condition.

## Claims

### Claim 1

A computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising:
synthesizing first domain audio data from first domain text data from a first domain;
feeding the synthesized first domain audio data into a trained encoder of the recurrent neural network transducer (RNN-T) having an initial condition, and updating the encoder by inputting both the synthesized first domain audio data and the first domain text data;
synthesizing second domain audio data from second domain text data from a second domain;
feeding the synthesized second domain audio data into the updated encoder of the recurrent neural network transducer (RNN-T), and updating a corresponding prediction network by inputting both the synthesized second domain audio data and the second domain text data;
encoding a sequence of labels from the first and second domain text data into a text embedding using the prediction network, and combining outputs of the encoder and the prediction network with the synthesized first and second domain audio data using a joiner; and
resetting the updated encoder to a pre-customized state by restoring weights on the updated encoder to the initial condition including a state trained on the first domain.

### Claim 2 (depends on 1)

The method of claim 1, wherein the combining using the joiner produces an output as an induced local field, zt,u, that is fed into a softmax function.

### Claim 3 (depends on 2)

The method of claim 2, wherein the softmax function generates a posterior probability, P(y|t,u).

### Claim 4 (depends on 3)

The method of claim 3, therein the posterior, probability generator P(y|t,u) generates an output that is an output sequence y=(y1, y2, . . . yU−1, yU) that is a length U output sequence based on an input feature sequence, x, that is a time-ordered sequence of acoustic features represented as vectors.

### Claim 5 (depends on 4)

The method of claim 4, wherein the input feature sequence, x, is derived from the synthesized first domain audio data.

### Claim 6

A system for customizing a recurrent neural network transducer (RNN-T), comprising:
one or more processor devices;
a memory in communication with at least one of the one or more processor devices; and
a display screen;
wherein the memory includes:
an encoder configured to receive synthesized first domain audio data generated from first domain text data from a first domain, the encoder being a trained encoder of the recurrent neural network transducer (RNN-T) having an initial condition, is configured to be undated from the initial condition by inputting both the synthesized first domain audio data and the first domain text data, receive synthesized second domain audio data generated from second domain text data from a second domain and updating a corresponding prediction network by inputting both the synthesized second domain audio data and the second domain text data, encode a sequence of labels from the first and second domain text data into a text embedding using the prediction network, combine outputs of the encoder and the prediction network with the synthesized first and second domain audio data using a joiner, and a pre-customized state by restoring weights on the encoder to the initial condition including a state trained on the first domain; and
an output sequence generator that produces output symbol sequence, y, based on an input feature sequence, x, that is a time-ordered sequence of acoustic features represented as vectors.

### Claim 7 (depends on 6)

The system of claim 6, wherein the joiner produces an induced local field, Zt,u as the output.

### Claim 8 (depends on 7)

The system of claim 7, wherein the memory further includes a softmax function that is configured to receive induced local field and generate an output.

### Claim 9 (depends on 8)

The system of claim 8, wherein the output sequence y=(y1, y2, . . . yU−1, yU) is a length U output sequence based on an input feature sequence, x, that is a time-ordered sequence of acoustic features represented as vectors.

### Claim 10 (depends on 9)

The system of claim 9, wherein the memory further includes a synthesizer that is configured to synthesize first domain audio data from first domain text data, and to synthesize second domain audio data from second domain text data.

### Claim 11

A computer program product for customizing a recurrent neural network transducer (RNN-T), the computer program product comprising one or more computer readable storage media, and program in: collectively stored on the one or more computer readable storage media; the program instructions executable by a computer, to cause the computer to:
synthesize first domain audio data from first domain text data from a first domain;
feed the synthesized first domain audio data into a trained encoder of the recurrent neural network transducer (RN) having an initial condition, and updating the encoder by inputting both the synthesized first domain audio data and the first domain text data;
synthesize second domain audio data from second domain text data from a second domain;
feed the synthesized second domain audio data into the updated encoder of the recurrent neural network transducer (RNN-T), and updating a corresponding predictor by inputting both the synthesized second domain audio data and the second domain text data;
encode a sequence of labels from the first and second domain text data into a text embedding a using the predictor, and combine outputs of the encoder and the predictor with the synthesized first and second domain audio data using a joiner, and
reset the updated encoder to a pre-customized state by restoring weights on the updated encoder to the initial condition including a state trained on the first domain.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein the combining using, the joiner produces an output as an induced local field, Zt,u that is fed into a softmax function.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the softmax function generates a posterior probability, P(y|t,u).

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein the posterior probability generator P(y|u) generates an output that is an output sequence y=(y1, y2, . . . yU−1, yU) that is a length U output sequence based on an input feature sequence, x, that is a time-ordered sequence of acoustic features represented as vectors.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein the first domain audio data is the input feature sequence, x.

### Claim 16

A computer-implemented method for customizing a recurrent neural network transducer (RNN-T), comprising:
synthesizing first domain audio data from first domain text data from a first domain;
feeding the synthesized first domain audio data into a trained encoder of the recurrent neural network transducer (RNN-T) having an initial condition, and updating the encoder by inputting both the synthesized first domain audio data and the first domain text data by encoding the synthesized first domain audio data into acoustic embedding, a1, wherein the acoustic embedding, a1, compresses the synthesized first domain audio data into a smaller feature space;
feeding the acoustic, embedding, a1, to a joiner;
synthesizing second domain audio data from second domain text data from a second domain;
feeding the synthesized second domain audio data into the updated encoder, wherein the updated encoder encodes the synthesized second domain audio data into acoustic embedding, b1, wherein the acoustic embedding, b1, compresses the synthesized second domain audio data into a smaller feature space;
feeding an output sequence from the joiner into a predictor of the recurrent neural network transducer (RNN-T), and updating the predictor by inputting both the synthesized second domain audio data and the second domain text data;
encoding a sequence of labels from the ins, and second domain text data into a text embedding using the predictor, and combining outputs of the encoder and the predictor with the synthesized first and second domain audio data using a joiner; and
resetting the updated encoder to a pre-customized state by restoring weights on the updated encoder to the initial condition including a state trained on the first domain.

### Claim 17 (depends on 16)

The method of claim 16, wherein the joiner combines the acoustic embedding, at, with an embedding from the predictor through a weighted summation.

### Claim 18 (depends on 17)

The method of claim 17, wherein the joiner produces an output as an induced local field, that is fed into a softmax function.

### Claim 19 (depends on 18)

The method of claim 18, wherein the softmax function generates a posterior probability, P(y|t,u).