# US 10,482,373 B1 · Grid long short-term memory neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,373 B1
- Document ID: US10482373B1
- Status: Granted
- Kind: B1
- Application number: 15/174,806
- Filed: 2016-06-06
- Granted: 2019-11-19
- Earliest claimed: 2015-06-05
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Nal Emmerich Kalchbrenner; Ivo Danihelka; Alexander Benjamin Graves
- Web page: https://patentgrep.com/patent/US10482373B1

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing grid Long Short-Term Memory (LSTM) neural networks that includes a plurality of N-LSTM blocks arranged in an N-dimensional grid. Each N-LSTM block is configured to: receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions; receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and, for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.

## Claims

### Claim 1

A system for processing a neural network input to generate a neural network output, the system comprising:
a grid Long Short-Term Memory (LSTM) neural network implemented by one or more computers, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein each N-LSTM block is configured to:
receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.

### Claim 2 (depends on 1)

The system of claim 1, wherein for each of the dimensions, applying a respective transform comprises:
generating a concatenated hidden vector for the dimension; and
applying the respective transform for the dimension to the concatenated hidden vector for the dimension.

### Claim 3 (depends on 2)

The system of claim 2, wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the N input hidden vectors.

### Claim 4 (depends on 2)

The system of claim 2, wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the input hidden vector for the dimension and the new hidden vectors for each other dimension.

### Claim 5 (depends on 2)

The system of claim 2, wherein, for one or more of the dimensions, applying the respective transform comprises applying a LSTM transform.

### Claim 6 (depends on 2)

The system of claim 2, wherein, for one or more of the dimensions, applying the respective transform comprises applying a non-LSTM transform.

### Claim 7 (depends on 1)

The system of claim 1, wherein, for a first N-LSTM block of the plurality of N-LSTM blocks, the input hidden vector and the input memory vector corresponding to a particular dimension are at least a portion of the neural network input.

### Claim 8 (depends on 7)

The system of claim 7, wherein the grid LSTM neural network further comprises an output layer configured to:
process the new hidden vector and the new memory vector corresponding to the particular dimension and generated by a second N-LSTM block of the plurality of N-LSTM blocks to generate the neural network output.

### Claim 9 (depends on 1)

The system of claim 1, wherein N is 1.

### Claim 10 (depends on 1)

The system of claim 1, wherein N is 2.

### Claim 11 (depends on 1)

The system of claim 1, wherein N is 3.

### Claim 12 (depends on 1)

The system of claim 1, wherein N is greater than 1.

### Claim 13

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:
a grid Long Short-Term Memory (LSTM) neural network, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein each N-LSTM block is configured to:
receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.

### Claim 14 (depends on 13)

The one or more non-transitory computer storage media of claim 13, wherein for each of the dimensions, applying a respective transform comprises:
generating a concatenated hidden vector for the dimension; and
applying the respective transform for the dimension to the concatenated hidden vector for the dimension.

### Claim 15 (depends on 14)

The one or more non-transitory computer storage media of claim 14, wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the N input hidden vectors.

### Claim 16 (depends on 14)

The one or more non-transitory computer storage media of claim 14, wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the input hidden vector for the dimension and the new hidden vectors for each other dimension.

### Claim 17 (depends on 14)

The one or more non-transitory computer storage media of claim 14, wherein, for one or more of the dimensions, applying the respective transform comprises applying a LSTM transform.

### Claim 18 (depends on 13)

The one or more non-transitory computer storage media of claim 13, wherein, for a first N-LSTM block of the plurality of N-LSTM blocks, the input hidden vector and the input memory vector corresponding to a particular dimension are at least a portion of a neural network input.

### Claim 19 (depends on 18)

The one or more non-transitory computer storage media of claim 18, wherein the grid LSTM neural network further comprises an output layer configured to:
process the new hidden vector and the new memory vector corresponding to the particular dimension and generated by a second N-LSTM block of the plurality of N-LSTM blocks to generate a neural network output.

### Claim 20

A method for processing a neural network input to generate a neural network output by processing the neural network input using a grid Long Short-Term Memory (LSTM) neural network, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein the method comprises, for each N-LSTM block:
receiving N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receiving N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, applying a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.