# US 2018/0025257 A1 · Generating images using neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025257 A1
- Document ID: US20180025257A1
- Status: Published application
- Kind: A1
- Application number: 15/721,089
- Filed: 2017-09-29
- Published: 2018-01-25
- Earliest claimed: 2016-01-25
- Assignees: DeepMind Technologies Limited
- Applicants: Google Inc.
- Inventors: Aaron Gerard Antonius van den Oord; Nal Emmerich Kalchbrenner; Karen Simonyan
- Web page: https://patentgrep.com/patent/US20180025257A1

## CPC

- H04N 19/50: using predictive coding (H04N19/61 takes precedence)
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/56: relating to colour
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- H04N 19/52: by predictive encoding
- H04N 19/172: the region being a picture, frame or field
- H04N 19/182: the unit being a pixel
- H04N 19/186: the unit being a colour or a chrominance component
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating images using neural networks. One of the methods includes generating the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels.

## Claims

### Claim 1

A neural network system implemented by one or more computers, the neural network system being configured to receive a neural network input and to generate an output image from the neural network input, the output image comprising a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels, and the neural network system comprising:
one or more initial neural network layers configured to receive the neural network input and to process the neural network input to generate an alternative representation of the neural network input; and
one or more output layers, wherein the output layers are configured to receive the alternative representation and to generate the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels.

### Claim 2 (depends on 1)

The neural network system of claim 1, wherein the plurality of color channels are ordered, wherein the one or more output layers comprise a respective output layer corresponding to each of the plurality of color channels, and wherein each of the output layers is configured to, for each pixel of the output image:
generate the respective score distribution over the discrete set of possible color values for the color channel corresponding to the output layer conditioned on (i) color values for pixels for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel corresponding to the output layer in the order of color channels.

### Claim 3 (depends on 2)

The neural network system of claim 2, wherein, for each pixel, each of the output layers is configured to receive a portion of the alternative representation corresponding to the color channel and context information based on (i) color values for pixels for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel corresponding to the output layer in the order of color channels.

### Claim 4 (depends on 3)

The neural network system of claim 3, wherein the neural network system is configured to apply a mask to outputs of neural network layers in the one or more initial neural network layers such that the portion of the alternative representation corresponding to the color channel is only generated based on pixels before the pixel in the sequence and color channel data for the pixel for color channels before the color channel corresponding to the output layer in the order of color channels.

### Claim 5 (depends on 1)

The neural network system of claim 1,
wherein the neural network input is an image,
wherein the one or more initial neural network layers comprise a row long short-term memory (LSTM) layer, and wherein the row LSTM layer is configured to:
process an input image row by row from top to bottom computing features for a whole row at once.

### Claim 6 (depends on 5)

The neural network system of claim 5, wherein the row LSTM layer computes the features using a one-dimensional convolution.

### Claim 7 (depends on 5)

The neural network system of claim 5, wherein the row LSTM layer is configured to:
compute an input-to-state component of the row LSTM layer for the entire input image; and
after computing the input-to-state component for the entire input image, use the input-to-state component in processing the input image row by row from top to bottom computing features for a whole row at once.

### Claim 8 (depends on 1)

The neural network system of claim 1,
wherein the neural network input is an image,
wherein the one or more initial neural network layers comprise a diagonal bidirectional LSTM (BiLSTM) layer, and wherein the diagonal BiLSTM layer is configured to:
scan an input image map in diagonal fashion along a first direction and in diagonal fashion along a second direction to generate features of the input image map.

### Claim 9 (depends on 8)

The neural network system of claim 8, wherein the diagonal BiLSTM layer is configured to:
skew the input image map into a space that allows for easily applying convolutions along diagonals; and
for each of the first direction and the second direction:
compute an input-to-state component of the diagonal biLSTM layer for the direction by applying a 1×1 convolution to the skewed input image map; and
compute a state-to-state recurrent component of the diagonal biLSTM layer for the direction by applying a column-wise convolution to the skewed input image map.

### Claim 10 (depends on 1)

The neural network system of claim 1, wherein the input is an image, and wherein the one or more initial neural network layers comprise one or more convolutional neural network layers.

### Claim 11

A system for generating a predicted next video frame that is a prediction of a video frame that will follow a last video frame in a temporal sequence of video frames, wherein the predicted next video frame comprises a plurality of pixels each having a respective color value for each of a plurality of color channels, wherein the pixels are ordered according to a pixel order and the color channels are ordered according to a channel order, and wherein the system comprises:
an encoder neural network implemented by one or more computers and configured to:
process each video frame in the temporal sequence to generate an encoded representation of the temporal sequence of video frames; and
a decoder neural network implemented by the one or more computers and configured to, for each color channel of each pixel in the predicted next video frame:
receive an input comprising (i) the encoded representation of the temporal sequence of video frames, (ii) color values for any pixels in the predicted next video frame before the pixel in the pixel order, and (iii) color values for the pixel for any color channels before the color channel in the channel order; and
process the input to generate an output defining a score distribution over a plurality of possible color values for the color channel of the pixel.

### Claim 12 (depends on 11)

The system of claim 11, further comprising:
a decoder subsystem configured to generate the predicted next video frame pixel by pixel according to the pixel order and the channel order by, for each color channel of each pixel,
providing as input to the decoder neural network (i) the encoded representation of the temporal sequence of video frames, (ii) color values for any pixels in the predicted next video frame before the pixel in the pixel order, and (iii) color values for the pixel for any color channels before the color channel in the channel order; and
determining the color value for the color channel of the pixel by sampling from the score distribution defined by the output of the decoder neural network.

### Claim 13 (depends on 11)

The system of claim 11, wherein the encoder neural network comprises:
a convolutional sub-neural network configured to, for each video frame in the temporal sequence, process the video frame to generate a convolved representation of the video frame that preserves a resolution of the video frame; and
a convolutional LSTM sub-neural network configured to, for each video frame in the temporal sequence, process the convolved representation to generate an encoded representation that preserves the resolution of the video frames in the temporal sequence.

### Claim 14 (depends on 13)

The system of claim 13, wherein the convolutional sub-neural network comprises a plurality of residual blocks of convolutional neural network layers.

### Claim 15 (depends on 14)

The system of claim 14, wherein the residual blocks are residual multiplicative blocks.

### Claim 16 (depends on 14)

The system of claim 14, wherein the convolutional neural network layers comprise one or more dilated convolutional neural network layers.

### Claim 17 (depends on 11)

The system of claim 11, wherein the decoder neural network comprises a plurality of residual blocks of convolutional neural network layers and an output layer.

### Claim 18 (depends on 16)

The system of claim 16, wherein the convolutional neural network layers comprise one or more masked convolutional neural network layers.

### Claim 19 (depends on 11)

The system of claim 11,
wherein the video frames in the temporal sequence are video frames of an environment being interacted with by a robotic agent, wherein each video frame is associated with a respective action vector defining an action to be performed by the robotic agent when the environment is in the state characterized by the associated video frame, and wherein, when generating the predicted next frame, the decoder neural network is conditioned on the action vector defining an action to be performed by the robotic agent when the environment is in the state characterized by the last video frame in the temporal sequence.

### Claim 20 (depends on 18)

The system of claim 18, wherein the encoder neural network is conditioned on action vectors and, optionally, on state vectors characterizing states of the robotic agent.