# US 10,319,374 B2 · Deployed end-to-end speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,374 B2
- Document ID: US10319374B2
- Status: Granted
- Kind: B2
- Application number: 15/358,083
- Filed: 2016-11-21
- Published: 2017-05-25
- Granted: 2019-06-11
- Earliest claimed: 2015-11-25
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Bryan Catanzaro; Jingdong Chen; Mike Chrzanowski; Erich Elsen; Jesse Engel; Christopher Fougner; Xu Han; Awni Hannun; Ryan Prenger; Sanjeev Satheesh; Shubhabrata Sengupta; Dani Yogatama; Chong Wang; Jun Zhan; Zhenyao Zhu; Dario Amodei
- Web page: https://patentgrep.com/patent/US10319374B2

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence)
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/21: the extracted parameters being power information
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments of end-to-end deep learning systems and methods are disclosed to recognize speech of vastly different languages, such as English or Mandarin Chinese. In embodiments, the entire pipelines of hand-engineered components are replaced with neural networks, and the end-to-end learning allows handling a diverse variety of speech including noisy environments, accents, and different languages. Using a trained embodiment and an embodiment of a batch dispatch technique with GPUs in a data center, an end-to-end deep learning system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

## Claims

### Claim 1

A computer-implemented method for transcribing speech comprising:
receiving an input audio from a user, the input audio comprising an utterance;
generating a set of spectrogram frames for the utterance;
inputting the set of spectrogram frames into a deep neural network (DNN) model, the DNN model comprising one or more convolution layers, one or more recurrent layers, and a row convolution layer in which an activation at a current time step of the row convolution layer is obtained using information from at least one of the one or more recurrent layers at the current time step and at least one future time step, the DNN model having been trained using a plurality of minibatches of training utterance sequences and batch normalization to normalize pre-activations in at least one of the one or more recurrent layers during training;
obtaining probabilities for one or more predicted characters from the DNN model; and
performing a search to obtain a transcription of the utterance using at least some of the probabilities of the one or more predicted characters and a language model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the batch normalization comprises computing a mean and variance over the length of each training utterance sequence in each minibatch for each hidden unit in one or more of the one or more recurrent layers.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein the row convolution layer is functionally positioned above the one or more recurrent layers.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3 wherein the one or more recurrent layers are unidirectional and forward-only layers.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the activation of the row convolution layer is used for character prediction corresponding to the current time step.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1 wherein the predicted characters are English characters or Chinese characters.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 wherein the input audio is normalized to make a total power of the input audio consistent with a set of training samples used to train the DNN model.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein a beam search in the language model is implemented to consider characters with a cumulative probability having at least a threshold probability.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1 wherein in generating the set of spectrogram frames, subsampling the utterance is implemented in obtaining the set of spectrogram frames by taking strides of a step size of predetermined number of time slices.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1 wherein the predicted characters from the DNN model comprise non-overlapping n-grams formed from words.

### Claim 11

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps to be performed comprising:
receiving an input audio, the input audio comprising an utterance;
obtaining a set of spectrogram frames for the utterance;
inputting the set of spectrogram frames into a trained neural network, the trained neural network comprising one or more convolution layers, one or more recurrent layers, and a row convolution layer in which an activation at a current time step of the row convolution layer is obtained using information from at least one of the one or more recurrent layers at the current time step and at least one future time;
obtaining probabilities of one or more predicted characters from the trained neural network; and
obtaining a transcription of the utterance using probabilities constrained by a language model.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium or media of claim 11 wherein the step of inputting the set of spectrogram frames into a trained neural network comprises taking strides of a step size of predetermined number of time slices.

### Claim 13 (depends on 11)

The non-transitory computer-readable medium or media of claim 11 wherein the one or more predicted character are selected from a model alphabet comprising the English alphabet and symbols representing alternate labellings comprising non-overlapping n-grams.

### Claim 14 (depends on 11)

The non-transitory computer-readable medium or media of claim 11 wherein the steps further comprising normalizing the input audio by using statistics from the training data set.

### Claim 15 (depends on 11)

The non-transitory computer-readable medium or media of claim 11 wherein the one or more recurrent layers are unidirectional and forward-only.

### Claim 16

A computer-implemented method for speech transcription, the method comprising:
receiving a set of spectrogram frames which correspond to an utterance;
using the set of spectrogram frames as input to a trained neural network, which comprises one or more recurrent layers, whose activation at a current time step is determined using a hidden state of a recurrent layer from the one or more recurrent layers at the current time step and a future hidden states context comprising hidden states of the recurrent layer for one or more future time steps, to obtain one or more predicted characters corresponding to the current time step.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16 wherein the one or more recurrent layers are forward-only layers.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16 wherein the one or more recurrent layers are trained using a plurality of minibatches of training utterance sequences from a training data set, the plurality of minibatches being normalized during training to normalize pre-activations in at least one of the one or more recurrent layers.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16 wherein the activation at the current time step of the row convolution layer is obtained from a weighted sum of a feature matrix comprising the hidden state of the recurrent layer and the future hidden states context of the recurrent layer, in which values of the feature matrix are weighted by a parameter matrix.

### Claim 20 (depends on 16)

The computer-implemented method of claim 16 further comprising the step of obtaining a transcription of the utterance using at least some of the predicted characters of the time steps and a language model.