# US 2020/0286464 A1 · Feature and feature variant reconstruction for recurrent model accuracy improvement in speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286464 A1
- Document ID: US20200286464A1
- Status: Published application
- Kind: A1
- Application number: 16/297,052
- Filed: 2019-03-08
- Published: 2020-09-10
- Earliest claimed: 2019-03-08
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Gakuto Kurata; Kartik Audhkhasi
- Web page: https://patentgrep.com/patent/US20200286464A1

## CPC

- G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A multi-task learning system is provided for speech recognition. The system includes a common encoder network. The system further includes a primary network for minimizing a Connectionist Temporal Classification (CTC) loss for speech recognition. The system also includes a sub network for minimizing a Mean squared error (MSE) loss for feature reconstruction. A first set of output data of the common encoder network is received by both of the primary network and the sub network. A second set of the output data of the common encode network is received only by the primary network from among the primary network and the sub network.

## Claims

### Claim 1

A multi-task learning system for speech recognition, comprising:
a common encoder network;
a primary network for minimizing a Connectionist Temporal Classification (CTC) loss for speech recognition; and
a sub network for minimizing a Mean squared error (MSE) loss for feature reconstruction, wherein a first set of output data of the common encoder network is received by both of the primary network and the sub network, and wherein a second set of the output data of the common encode network is received only by the primary network from among the primary network and the sub network.

### Claim 2 (depends on 1)

The multi-task learning system of claim 1, wherein, for the first set of output data, the CTC loss is minimized after the MSE loss is minimized.

### Claim 3 (depends on 1)

The multi-task learning system of claim 1, wherein, for the second set of output data, only the CTC loss is minimized.

### Claim 4 (depends on 1)

The multi-task learning system of claim 1, wherein for decoding an acoustic utterance from which the input sequence is derived, an output of the primary network is used while an output of the sub network is discarded.

### Claim 5 (depends on 1)

The multi-task learning system of claim 1, wherein an input sequence which the common encoder network receives is distorted when minimizing the MSE loss by swapping the former and latter parts of the input sequence.

### Claim 6 (depends on 1)

The multi-task learning system of claim 1, wherein an input sequence which the common encoder network receives is distorted when minimizing the MSE loss by stripping a beginning portion and an end portion of the sequence for the feature reconstruction.

### Claim 7 (depends on 1)

The multi-task learning system of claim 1, wherein the common encoder network is common to both the primary network and the sub network.

### Claim 8 (depends on 1)

The multi-task learning system of claim 1, wherein the common encoder network is a bi-directional Long Short Term Memory network.

### Claim 9 (depends on 1)

The multi-task learning system of claim 1, wherein the sub network is a bi-directional Long Short Term Memory network.

### Claim 10 (depends on 1)

The multi-task learning system of claim 1, wherein the first set of output data of the common encoder network is randomly selected.

### Claim 11

A computer-implemented method for speech recognition, comprising:
outputting, by a common encoder network, a first set of output data and a second set of output data;
minimizing, by a primary network, a Connectionist Temporal Classification (CTC) loss for speech recognition; and
minimizing, by a sub network, a Mean squared error (MSE) loss for feature reconstruction, wherein a first set of output data of the common encoder network is received by both of the primary network and the sub network, and wherein a second set of the output data of the common encode network is received only by the primary network from among the primary network and the sub network.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein, for the first set of output data, the CTC loss is minimized after the MSE loss is minimized.

### Claim 13 (depends on 11)

The computer-implemented method of claim 11, wherein, for the second set of output data, only the CTC loss is minimized.

### Claim 14 (depends on 11)

The computer-implemented method of claim 11, wherein for decoding an acoustic utterance from which the input sequence is derived, an output of the primary network is used while an output of the sub network is discarded.

### Claim 15 (depends on 11)

The computer-implemented method of claim 11, wherein an input sequence which the common encoder network receives is distorted when minimizing the MSE loss by swapping the former and latter parts of the input sequence for the feature reconstruction.

### Claim 16 (depends on 11)

The computer-implemented method of claim 11, wherein an input sequence which the common encoder network receives is distorted when minimizing the MSE loss by stripping a beginning portion and an end portion of the sequence for the feature reconstruction.

### Claim 17 (depends on 11)

The computer-implemented method of claim 11, wherein the common encoder network is common to both the primary network and the sub network.

### Claim 18 (depends on 11)

The computer-implemented method of claim 11, wherein the common encoder network is a bi-directional Long Short Term Memory network.

### Claim 19 (depends on 11)

The computer-implemented method of claim 11, wherein the sub network is a bi-directional Long Short Term Memory network.

### Claim 20

A computer program product for speech recognition, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
outputting, by a common encoder network of the computer, a first set of output data and a second set of output data;
minimizing, by a primary network of the computer, a Connectionist Temporal Classification (CTC) loss for speech recognition; and
minimizing, by a sub network of the computer, a Mean squared error (MSE) loss for feature reconstruction, wherein a first set of output data of the common encoder network is received by both of the primary network and the sub network, and wherein a second set of the output data of the common encode network is received only by the primary network from among the primary network and the sub network.
