# US 2018/0025721 A1 · Automatic speech recognition using multi-dimensional models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025721 A1
- Document ID: US20180025721A1
- Status: Published application
- Kind: A1
- Application number: 15/217,457
- Filed: 2016-07-22
- Published: 2018-01-25
- Earliest claimed: 2016-07-22
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Bo Li; Tara N. Sainath
- Web page: https://patentgrep.com/patent/US20180025721A1

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatic speech recognition using multi-dimensional models. In some implementations, audio data that describes an utterance is received. A transcription for the utterance is determined using an acoustic model that includes a neural network having first memory blocks for time information and second memory blocks for frequency information. The transcription for the utterance is provided as output of an automated speech recognizer.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
receiving, by the one or more computers, audio data that describes an utterance;
determining, by the one or more computers, a transcription for the utterance, comprising processing the audio data using a neural network that has been trained as an acoustic model, the neural network having first memory blocks for time information and second memory blocks for frequency information, the first memory blocks being different from the second memory blocks, wherein the first memory blocks are time-LSTM blocks that each have a state, and wherein the second memory blocks are frequency-LSTM blocks that each have a state and a corresponding frequency step in a sequence of multiple frequency steps, wherein the states are determined for each of a sequence of multiple time steps;
wherein, for each of at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state using the state of the time-LSTM block corresponding to the same frequency step at the previous time step; and
wherein, for each of at least some of the time-LSTM blocks, the time-LSTM block determines its state using the state of the frequency-LSTM block corresponding to the same time step and the previous frequency step; and
providing, by the one or more computers, the transcription for the utterance as output of an automated speech recognizer.

### Claim 2 (depends on 1)

The method of claim 1, wherein receiving the audio data comprises receiving, over a network, audio data generated by a client device; and
wherein providing the transcription comprises providing, over the network, the transcription to the client device.

### Claim 3

(canceled)

### Claim 4 (depends on 1)

The method of claim 1, wherein the neural network comprises a grid-LSTM module, a linear projection layer, one or more LSTM layers, and a deep neural network (DNN);
wherein the grid-LSTM module includes the first memory blocks and the second memory blocks, and the grid-LSTM module provides output to the linear projection layer;
wherein the linear projection layer provides output to the one or more LSTM layers; and
wherein the one or more LSTM layers provide output to the DNN.

### Claim 5 (depends on 1)

The method of claim 1, wherein the neural network is configured to share information from each of the first memory blocks with a respective proper subset of the second memory blocks, and the neural network is configured to share information from each of the second memory blocks with a respective proper subset of the first memory blocks.

### Claim 6

(canceled)

### Claim 7 (depends on 1)

The method of claim 1, wherein, for at least some of the time-LSTM blocks, the time-LSTM block determines its state based on (i) input received for a current time step, (ii) the state of the time-LSTM block at a previous time step, and (iii) a state of exactly one of the frequency-LSTM blocks; and
wherein, for at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state based on (i) input received for a current time step, (ii) the state of the frequency-LSTM block for a previous frequency step at the current time step, and (iii) a state of exactly one of the time-LSTM blocks.

### Claim 8 (depends on 1)

The method of claim 1, wherein each of the time-LSTM blocks and the frequency-LSTM blocks has one or more weights, and wherein the weights for the time-LSTM blocks are independent of the weights for the frequency-LSTM blocks.

### Claim 9 (depends on 1)

The method of claim 1, wherein each of the time-LSTM blocks and the frequency-LSTM blocks has one or more weights, and wherein at least some of the weights are shared between the time-LSTM blocks and the frequency-LSTM blocks.

### Claim 10

(canceled)

### Claim 11 (depends on 1)

The method of claim 1, wherein processing the audio data using the neural network comprises:
providing, as input to the neural network, input vectors having values describing the utterance; and
receiving, as output of the neural network, one or more outputs that each indicate a likelihood that a respective phonetic unit represents a portion of the utterance.

### Claim 12 (depends on 11)

The method of claim 11, wherein receiving the one or more outputs comprises receiving, as output of the neural network, multiple outputs corresponding to different context-dependent states of phones, wherein each of the multiple outputs indicates a likelihood of occurrence for the corresponding context-dependent state.

### Claim 13 (depends on 11)

The method of claim 11, wherein providing the input vectors comprises providing input vectors comprising values for log-mel features.

### Claim 14 (depends on 11)

The method of claim 11, wherein providing the input vectors comprises providing values representing audio waveform features;
wherein the audio waveform features are determined using a filterbank having parameters trained jointly with weights of the neural network.

### Claim 15 (depends on 11)

The method of claim 11, wherein providing the input vectors comprises providing values representing characteristics of multiple channels of audio describing the utterance.

### Claim 16

A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving, by the one or more computers, audio data that describes an utterance;
determining, by the one or more computers, a transcription for the utterance, comprising processing the audio data using a neural network that has been trained as an acoustic model, the neural network having first memory blocks for time information and second memory blocks for frequency information, the first memory blocks being different from the second memory blocks, wherein the first memory blocks are time-LSTM blocks that each have a state, and wherein the second memory blocks are frequency-LSTM blocks that each have a state and a corresponding frequency step in a sequence of multiple frequency steps, wherein the states are determined for each of a sequence of multiple time steps;
wherein, for each of at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state using the state of the time-LSTM block corresponding to the same frequency step at the previous time step; and
wherein, for each of at least some of the time-LSTM blocks, the time-LSTM block determines its state using the state of the frequency-LSTM block corresponding to the same time step and the previous frequency step; and
providing, by the one or more computers, the transcription for the utterance as output of an automated speech recognizer.

### Claim 17

(canceled)

### Claim 18 (depends on 16)

The system of claim 16, wherein the neural network comprises a grid-LSTM module, a linear projection layer, one or more LSTM layers, and a deep neural network (DNN);
wherein the grid-LSTM module includes the first memory blocks and the second memory blocks, and the grid-LSTM module provides output to the linear projection layer;
wherein the linear projection layer provides output to the one or more LSTM layers; and
wherein the one or more LSTM layers provide output to the DNN.

### Claim 19

(canceled)

### Claim 20

One or more non-transitory computer-readable storage media storing a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving, by the one or more computers, audio data that describes an utterance;
determining, by the one or more computers, a transcription for the utterance, comprising processing the audio data using a neural network that has been trained as an acoustic model, the neural network having first memory blocks for time information and second memory blocks for frequency information, the first memory blocks being different from the second memory blocks, wherein the first memory blocks are time-LSTM blocks that each have a state, and wherein the second memory blocks are frequency-LSTM blocks that each have a state and a corresponding frequency step in a sequence of multiple frequency steps, wherein the states are determined for each of a sequence of multiple time steps;
wherein, for each of at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state using the state of the time-LSTM block corresponding to the same frequency step at the previous time step; and
wherein, for each of at least some of the time-LSTM blocks, the time-LSTM block determines its state using the state of the frequency-LSTM block corresponding to the same time step and the previous frequency step; and
providing, by the one or more computers, the transcription for the utterance as output of an automated speech recognizer.

### Claim 21 (depends on 1)

The method of claim 1, wherein processing the audio data using the neural network comprises providing, as input to the neural network, values representing audio waveform features; and
wherein generating the transcription comprises applying filters having parameters trained jointly with weights of the neural network.

### Claim 22 (depends on 1)

The method of claim 1, comprising receiving, from the neural network, outputs indicative of probabilities of occurrence of linguistic units, the outputs being determined by processing the audio data using the neural network.

### Claim 23 (depends on 1)

The method of claim 1, wherein processing the audio data using the neural network comprises processing audio data representing characteristics of multiple channels of audio describing the utterance.

### Claim 24 (depends on 16)

The system of claim 16, wherein processing the audio data using the neural network comprises processing audio data representing characteristics of multiple channels of audio describing the utterance.

### Claim 25 (depends on 20)

The one or more non-transitory computer-readable storage media of claim 20, wherein processing the audio data using the neural network comprises processing audio data representing characteristics of multiple channels of audio describing the utterance.
