# US 10,373,610 B2 · Systems and methods for automatic unit selection and target decomposition for sequence labelling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,610 B2
- Document ID: US10373610B2
- Status: Granted
- Kind: B2
- Application number: 15/698,593
- Filed: 2017-09-07
- Published: 2018-08-30
- Granted: 2019-08-06
- Earliest claimed: 2017-02-24
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Hairong Liu; Zhenyao Zhu; Sanjeev Satheesh
- Web page: https://patentgrep.com/patent/US10373610B2

## CPC

- G10L 15/063: {Training}
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/16: using artificial neural networks
- G10L 15/187: Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 2015/0636: {Threshold criteria for the updating}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Described herein are systems and methods for automatic unit selection and target decomposition for sequence labelling. Embodiments include a new loss function called Gram-Connectionist Temporal Classification (CTC) loss that extend the popular CTC loss function criterion to alleviate prior limitations. While preserving the advantages of CTC, Gram-CTC automatically learns the best set of basic units (grams), as well as the most suitable decomposition of target sequences. Unlike CTC, embodiments of Gram-CTC allow a model to output variable number of characters at each time step, which enables the model to capture longer term dependency and improves the computational efficiency. It is also demonstrated that embodiments of Gram-CTC improve CTC in terms of both performance and efficiency on the large vocabulary speech recognition task at multiple scales of data, and that systems that employ an embodiment of Gram-CTC can outperform the state-of-the-art on a standard speech benchmark.

## Claims

### Claim 1

A computer-implemented method for training a transcription model for speech transcription, the method comprising:
inputting a sequence of utterances into the transcription model, with the utterances and associated labels sampled from a training set;
establishing a set of grams comprising uni-grams and high-order grams;
employing striding to process the input sequence of utterances with stride size depending on the largest gram of the gram set;
predicting, from the transcription model, one or more units for each utterance of the sequence of utterances, the one or more units selected from the established set of grams; and
implementing a Gram Connectionist Temporal Classification (Gram-CTC) loss function to measure an error of the predicted one or more units for each utterance given the associated labels, the Gram-CTC loss function automatically selects useful grams and decomposes the input sequence with the selected grams, the Gram-CTC loss function implementation involving forward-backward transitions;
computing a derivative of the loss with respect to parameters of the transcription model; and
updating the transcription model using the derivative through back-propagation.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the established set of grams comprises all uni-grams and bi-grams.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein the high-order grams are selected in a sample dataset with occurrence frequency above a threshold.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein predicted one or more units for each utterance comprises non-overlapping bigrams at word level.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein predicting one or more characters for each utterance comprises allowing decomposition of a word into large units with intuitive and meaningful high-order grams.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1 wherein predicting one or more characters for each utterance comprises allowing decomposition of a sentence into segments which are meaningful in term of pronunciation.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 further comprising training the transcription model jointly with a vanilla CTC loss function.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein the utterances have linear Fast Fourier Transform (FFT) features normalized such that each input feature has zero mean and unit variance.

### Claim 9

A computer-implemented method for training a recurrent neural network (RNN) model for speech transcription, the method comprising:
inputting a plurality of utterance and associated labels to the RNN model, the utterances and associated labels being sampled from a training set, the RNN model comprising one or more convolutional layers and one or more recurrent layers;
predicting one or more units among an established gram set through the one or more recurrent layers of the RNN model;
obtaining a probability distribution over the predicted units in an output layer of the RNN model; and
implementing a Gram Connectionist Temporal Classification (Gram-CTC) loss function to measure an error in prediction of a unit for the utterance given the associated label, the Gram-CTC loss function automatically selects useful grams and decomposes the input sequence with the selected grams;
computing a derivative of the loss with respect to parameters of the RNN model; and
updating the RNN model using the derivative through back-propagation.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9 wherein the one or more recurrent layers are forward or bidirectional Gated Recurrent layers.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9 wherein the output layer is a softmax layer.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9 wherein the one or more convolutional layers are two-dimensional convolutional layers.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9 wherein implementing the Gram-CTC loss function comprises forward and backward state transitions.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9 wherein the predicted one or more units are uni-grams or high-order grams.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9 wherein the established gram set comprises all uni-grams and selected high-order grams.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15 wherein the selected high-order grams are selected in a sample dataset with occurrence frequency above a threshold.

### Claim 17

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more microprocessors, causes the steps to be performed comprising:
receiving a plurality of utterance sequences, each utterance sequence and associated label being sampled from a training set;
outputting a probability distribution over predicted units corresponding to the utterance sequences to a Gram Connectionist Temporal Classification (Gram-CTC) loss function; the Gram-CTC loss function automatically selects useful grams among uni-grams and selected high-order grams to decomposes the utterance sequence with the selected grams; and
implementing a Gram-CTC loss function algorithm to measure an error of the predicted one or more characters for each utterance given the associated labels, the Gram-CTC loss function implementation involving forward-backward transitions.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium or media of claim 17 wherein the selected high-order grams are bi-grams.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium or media of claim 18 wherein the bi-grams selected in a sample dataset with occurrence frequency above a threshold.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium or media of claim 17 further comprising computing a derivative of the loss with respect to the predicted units and implementing back-propagation using the derivative.