Generating images using neural networks
AssigneeDeepMind Technologies Limited
InventorsAaron Gerard Antonius van den Oord, Nal Emmerich Kalchbrenner, Karen Simonyan
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating images using neural networks. One of the methods includes generating the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels.
Claims20
A system for generating a predicted next video frame that is a prediction of a video frame that will follow a last video frame in a temporal sequence of video frames, wherein the predicted next video frame comprises a plurality of pixels each having a respective color value for each of a plurality of color channels, wherein the pixels are ordered according to a pixel order and the color channels are ordered according to a channel order, and wherein the system comprises:
- an encoder neural network implemented by one or more computers and configured to:
- process each video frame in the temporal sequence to generate an encoded representation of the temporal sequence of video frames;
- and
- a decoder neural network implemented by the one or more computers and configured to, for each color channel of each pixel in the predicted next video frame:
- receive an input comprising (i) the encoded representation of the temporal sequence of video frames, (ii) color values for any pixels in the predicted next video frame before the pixel in the pixel order, and (iii) color values for the pixel for any color channels before the color channel in the channel order;
- and
- process the input to generate an output defining a score distribution over a plurality of possible color values for the color channel of the pixel.
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a system for generating a predicted next video frame that is a prediction of a video frame that will follow a last video frame in a temporal sequence of video frames, wherein the predicted next video frame comprises a plurality of pixels each having a respective color value for each of a plurality of color channels, wherein the pixels are ordered according to a pixel order and the color channels are ordered according to a channel order, and wherein the system comprises:
- an encoder neural network implemented by one or more computers and configured to:
- process each video frame in the temporal sequence to generate an encoded representation of the temporal sequence of video frames;
- and
- a decoder neural network implemented by the one or more computers and configured to, for each color channel of each pixel in the predicted next video frame:
- receive an input comprising (i) the encoded representation of the temporal sequence of video frames, (ii) color values for any pixels in the predicted next video frame before the pixel in the pixel order, and (iii) color values for the pixel for any color channels before the color channel in the channel order;
- and
- process the input to generate an output defining a score distribution over a plurality of possible color values for the color channel of the pixel.
A computer-implemented method of generating a predicted next video frame that is a prediction of a video frame that will follow a last video frame in a temporal sequence of video frames, wherein the predicted next video frame comprises a plurality of pixels each having a respective color value for each of a plurality of color channels, wherein the pixels are ordered according to a pixel order and the color channels are ordered according to a channel order, and wherein the method comprises:
- processing the temporal sequence of video frames using an encoder neural network configured to:
- process each video frame in the temporal sequence to generate an encoded representation of the temporal sequence of video frames;
- and
- processing the encoded representation using a decoder neural network configured to, for each color channel of each pixel in the predicted next video frame:
- receive an input comprising (i) the encoded representation of the temporal sequence of video frames, (ii) color values for any pixels in the predicted next video frame before the pixel in the pixel order, and (iii) color values for the pixel for any color channels before the color channel in the channel order;
- and
- process the input to generate an output defining a score distribution over a plurality of possible color values for the color channel of the pixel.
Dependencies are inferred from the claim text.