# US 10,770,064 B1 · System and method for speech recognition using deep recurrent neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,770,064 B1
- Document ID: US10770064B1
- Status: Granted
- Kind: B1
- Application number: 16/658,697
- Filed: 2019-10-21
- Granted: 2020-09-08
- Earliest claimed: 2012-11-29
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Alexander B. Graves
- Web page: https://patentgrep.com/patent/US10770064B1

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Deep recurrent neural networks applied to speech recognition. The deep recurrent neural networks (RNNs) are preferably implemented by stacked long short-term memory bidirectional RNNs. The RNNs are trained using end-to-end training with suitable regularisation.

## Claims

### Claim 1

A system for training
a deep recurrent neural network (“deep RNN”) to perform speech recognition, the deep RNN being an RNN comprising an input layer, an output layer, and multiple recurrent hidden layers organized in a stack on top of each other;
wherein the multiple recurrent hidden layers comprise:
a lowest hidden layer configured to receive input vectors from the input layer, and
a highest hidden layer configured to generate output vectors received as inputs by the output layer, wherein at least one of the multiple recurrent hidden layers is implemented by a Long Short-term Memory (“LSTM”) RNN, wherein the system comprises one or more computers storing instructions that when executed by the one or more computers cause one or more computers to perform operations comprising:
receiving training data comprising a plurality of sequences of audio observations and, for each sequence of audio observations, a corresponding sequence of text symbols that represents the sequence of audio observations; and
training the deep RNN to map the sequences of audio observations to the corresponding sequences of text symbols.

### Claim 2 (depends on 1)

The system of claim 1, wherein the training comprises training the deep RNN using Connectionist Temporal Classification (“CTC”).

### Claim 3 (depends on 2)

The system of claim 2, wherein the output layer is a softmax output layer that is configured to receive an output vector and to generate a respective score for each text symbol in a set of text symbols and a score for a blank symbol that represents a non-output.

### Claim 4 (depends on 2)

The system of claim 2, wherein the training operates without intermediate alignment or segmentation of training data and trains the network to map directly from input sequences to target sequences.

### Claim 5 (depends on 1)

The system of claim 1, wherein the training comprises training the deep RNN jointly with an RNN transducer that models a dependency of each text symbol on previous text symbols in the sequence of text symbols.

### Claim 6 (depends on 1)

The system of claim 1, wherein:
each hidden layer within the stack produces an output sequence of output vectors and receives an input sequence of input vectors; and
the output sequence of each hidden layer within the stack other than the highest hidden layer is the input sequence of the next higher hidden layer in the stack.

### Claim 7 (depends on 6)

The system of claim 6, wherein the same hidden layer function is used for all of the multiple recurrent hidden layers in the deep RNN.

### Claim 8 (depends on 1)

The system of claim 1, wherein one or more of the hidden layers is implemented by a bi-directional LSTM RNN.

### Claim 9

A method for training a deep recurrent neural network (“deep RNN”) to perform speech recognition, the deep RNN being an RNN comprising an input layer, an output layer, and multiple recurrent hidden layers organized in a stack on top of each other;
wherein the multiple recurrent hidden layers comprise:
a lowest hidden layer configured to receive input vectors from the input layer, and
a highest hidden layer configured to generate output vectors received as inputs by the output layer, wherein at least one of the multiple recurrent hidden layers is implemented by a Long Short-term Memory (“LSTM”) RNN, wherein the method comprises:
receiving training data comprising a plurality of sequences of audio observations and, for each sequence of audio observations, a corresponding sequence of text symbols that represents the sequence of audio observations; and
training the deep RNN to map the sequences of audio observations to the corresponding sequences of text symbols.

### Claim 10 (depends on 9)

The method of claim 9, wherein the training comprises training the deep RNN using Connectionist Temporal Classification (“CTC”).

### Claim 11 (depends on 10)

The method of claim 10, wherein the output layer is a softmax output layer that is configured to receive an output vector and to generate a respective score for each text symbol in a set of text symbols and a score for a blank symbol that represents a non-output.

### Claim 12 (depends on 10)

The method of claim 10, wherein the training operates without intermediate alignment or segmentation of training data and trains the network to map directly from input sequences to target sequences.

### Claim 13 (depends on 9)

The method of claim 9, wherein the training comprises training the deep RNN jointly with an RNN transducer that models a dependency of each text symbol on previous text symbols in the sequence of text symbols.

### Claim 14 (depends on 9)

The method of claim 9, wherein:
each hidden layer within the stack produces an output sequence of output vectors and receives an input sequence of input vectors; and
the output sequence of each hidden layer within the stack other than the highest hidden layer is the input sequence of the next higher hidden layer in the stack.

### Claim 15 (depends on 14)

The method of claim 14, wherein the same hidden layer function is used for all of the multiple recurrent hidden layers in the deep RNN.

### Claim 16 (depends on 9)

The method of claim 9, wherein one or more of the hidden layers is implemented by a bi-directional LSTM RNN.

### Claim 17

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a deep recurrent neural network (“deep RNN”) to perform speech recognition, the deep RNN being an RNN comprising an input layer, an output layer, and multiple recurrent hidden layers organized in a stack on top of each other;
wherein the multiple recurrent hidden layers comprise:
a lowest hidden layer configured to receive input vectors from the input layer, and
a highest hidden layer configured to generate output vectors received as inputs by the output layer, wherein at least one of the multiple recurrent hidden layers is implemented by a Long Short-term Memory (“LSTM”) RNN, wherein the operations comprise:
receiving training data comprising a plurality of sequences of audio observations and, for each sequence of audio observations, a corresponding sequence of text symbols that represents the sequence of audio observations; and
training the deep RNN to map the sequences of audio observations to the corresponding sequences of text symbols.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage media of claim 17, wherein the training comprises training the deep RNN using Connectionist Temporal Classification (“CTC”).

### Claim 19 (depends on 18)

The non-transitory computer-readable storage media of claim 18, wherein the output layer is a softmax output layer that is configured to receive an output vector and to generate a respective score for each text symbol in a set of text symbols and a score for a blank symbol that represents a non-output.

### Claim 20 (depends on 18)

The non-transitory computer-readable storage media of claim 18, wherein the training operates without intermediate alignment or segmentation of training data and trains the network to map directly from input sequences to target sequences.