# US 2023/0419076 A1 · Recurrent neural networks for data item generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419076 A1
- Document ID: US20230419076A1
- Status: Published application
- Kind: A1
- Application number: 18/367,305
- Filed: 2023-09-12
- Published: 2023-12-28
- Earliest claimed: 2015-02-06
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Karol Gregor; Ivo Danihelka
- Web page: https://patentgrep.com/patent/US20230419076A1

## CPC

- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 25/30: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods, and systems, including computer programs encoded on computer storage media for generating data items. A method includes reading a glimpse from a data item using a decoder hidden state vector of a decoder for a preceding time step, providing, as input to a encoder, the glimpse and decoder hidden state vector for the preceding time step for processing, receiving, as output from the encoder, a generated encoder hidden state vector for the time step, generating a decoder input from the generated encoder hidden state vector, providing the decoder input to the decoder for processing, receiving, as output from the decoder, a generated a decoder hidden state vector for the time step, generating a neural network output update from the decoder hidden state vector for the time step, and combining the neural network output update with a current neural network output to generate an updated neural network output.

## Claims

### Claim 1

(canceled)

### Claim 2

A computer implemented method for generating a final output image, comprising:
repeatedly updating a neural network output at each of a plurality of time steps to generate a final neural network output, the updating comprising, for each of the time steps:
generating a decoder input for a decoder neural network for the time step, wherein the decoder neural network is configured to, for each of the time steps, receive the decoder input for the time step, and process a decoder hidden state vector for a preceding time step and the decoder input to generate a decoder hidden state vector for the time step;
processing the decoder input for the time step using the decoder neural network to generate the decoder hidden state vector for the time step;
generating a neural network output update for the time step from the decoder hidden state vector for the time step; and
combining the neural network output update for the time step with a current neural network output to generate an updated neural network output; and
generating the final output image from the final neural network output.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the decoder input for the decoder neural network comprises selecting latent variable samples from a prior distribution.

### Claim 4 (depends on 2)

The method of claim 2, wherein the updated neural network outputs are images, and wherein the updated neural network output for a last time step in the plurality of time steps is a new image.

### Claim 5 (depends on 2)

The method of claim 2, wherein the updated neural network outputs define distributions of images, and wherein the updated neural network output for a last time step in the plurality of time steps defines a distribution that is used to parameterize a distribution model of the final output image.

### Claim 6 (depends on 2)

The method of claim 2, wherein generating the neural network output update for the time step from the decoder hidden state vector for the time step comprises:
applying a write operation to the decoder hidden state vector for the time step to generate the neural network output update for the time step.

### Claim 7 (depends on 6)

The method of claim 6, wherein applying the write operation comprises:
generating a set of attention parameters for the time step from the decoder hidden state vector for the time step; and
applying the set of attention parameters to the decoder hidden state vector for the time step to generate the neural network output update for the time step.

### Claim 8 (depends on 2)

The method of claim 2, wherein the neural network output update is a writing image patch.

### Claim 9 (depends on 2)

The method of claim 2, comprising, for each of the time steps:
reading a glimpse from an input image using a decoder hidden state vector of the decoder neural network for a preceding time step;
providing, as input to an encoder neural network, the (i) glimpse and (ii) the decoder hidden state vector for the preceding time step for processing;
receiving, as output from the encoder neural network, an encoder hidden state vector for the time step; and
generating the decoder input for the decoder neural network from the encoder hidden state vector at the time step.

### Claim 10 (depends on 9)

The method of claim 9, wherein generating the decoder input for the decoder neural network from the encoder hidden state vector for the time step comprises:
using the encoder hidden state vector for the time step to parameterize a distribution of a latent vector; and
sampling the decoder input from the distribution.

### Claim 11 (depends on 9)

The method of claim 9, wherein the encoder neural network and the decoder neural network have been trained to autoencode input images.

### Claim 12

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
repeatedly updating a neural network output at each of a plurality of time steps to generate a final neural network output, the updating comprising, for each of the time steps:
generating a decoder input for a decoder neural network for the time step, wherein the decoder neural network is configured to, for each of the time steps, receive the decoder input for the time step, and process a decoder hidden state vector for a preceding time step and the decoder input to generate a decoder hidden state vector for the time step;
processing the decoder input for the time step using the decoder neural network to generate the decoder hidden state vector for the time step;
generating a neural network output update for the time step from the decoder hidden state vector for the time step; and
combining the neural network output update for the time step with a current neural network output to generate an updated neural network output; and
generating a final output image from the final neural network output.

### Claim 13 (depends on 12)

The system of claim 12, wherein generating the decoder input for the decoder neural network comprises selecting latent variable samples from a prior distribution.

### Claim 14 (depends on 12)

The system of claim 12, wherein the updated neural network outputs are images, and wherein the updated neural network output for a last time step in the plurality of time steps is a new image.

### Claim 15 (depends on 12)

The system of claim 12, wherein the updated neural network outputs define distributions of images, and wherein the updated neural network output for a last time step in the plurality of time steps defines a distribution that is used to parameterize a distribution model of the final output image.

### Claim 16 (depends on 12)

The system of claim 12, wherein generating the neural network output update for the time step from the decoder hidden state vector for the time step comprises:
applying a write operation to the decoder hidden state vector for the time step to generate the neural network output update for the time step.

### Claim 17 (depends on 16)

The system of claim 16, wherein applying the write operation comprises:
generating a set of attention parameters for the time step from the decoder hidden state vector for the time step; and
applying the set of attention parameters to the decoder hidden state vector for the time step to generate the neural network output update for the time step.

### Claim 18 (depends on 12)

The system of claim 12, wherein the neural network output update is a writing image patch.

### Claim 19 (depends on 12)

The system of claim 12, the operations comprise, for each of the time steps:
reading a glimpse from an input image using a decoder hidden state vector of the decoder neural network for a preceding time step;
providing, as input to an encoder neural network, the (i) glimpse and (ii) the decoder hidden state vector for the preceding time step for processing;
receiving, as output from the encoder neural network, an encoder hidden state vector for the time step; and
generating the decoder input for the decoder neural network from the encoder hidden state vector at the time step.

### Claim 20 (depends on 19)

The system of claim 19, wherein generating the decoder input for the decoder neural network from the encoder hidden state vector for the time step comprises:
using the encoder hidden state vector for the time step to parameterize a distribution of a latent vector; and
sampling the decoder input from the distribution.

### Claim 21

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
repeatedly updating a neural network output at each of a plurality of time steps to generate a final neural network output, the updating comprising, for each of the time steps:
generating a decoder input for a decoder neural network for the time step, wherein the decoder neural network is configured to, for each of the time steps, receive the decoder input for the time step, and process a decoder hidden state vector for a preceding time step and the decoder input to generate a decoder hidden state vector for the time step;
processing the decoder input for the time step using the decoder neural network to generate the decoder hidden state vector for the time step;
generating a neural network output update for the time step from the decoder hidden state vector for the time step; and
combining the neural network output update for the time step with a current neural network output to generate an updated neural network output; and
generating a final output image from the final neural network output.
