Recurrent neural networks for data item generation

A computer implemented method for processing an input image through a recurrent encoder neural network and a recurrent decoder neural network to generate a final output image by repeatedly updating a neural network output at each of a plurality of time steps based on an input image to generate a final neural network output and generating the final output image from the final neural network output, the updating comprising, for each of the time step:

• reading a glimpse from the input image using a decoder hidden state vector of the decoder neural network for the preceding time step;
• providing, as input to the encoder neural network, the (i) glimpse and (ii) decoder hidden state vector for the preceding time step for processing;
• receiving, as output from the encoder neural network, a generated encoder hidden state vector for the time step;
• generating a decoder input for the decoder neural network from the generated encoder hidden state vector at the time step;
• providing, as input to the decoder neural network, the decoder input for processing;
• receiving, as output from the decoder neural network, a generated a decoder hidden state vector for the time step;
• generating a neural network output update for the time step from the decoder hidden state vector for the time step;
• and
• combining the neural network output update for the time step with a current neural network output to generate an updated neural network output.

A computer storage medium encoded with instructions that, when executed by one or more computers, cause one or more computers to implement a neural network system, the neural network system comprising:

• an encoder neural network, wherein the encoder neural network is a recurrent neural network that is configured to, for each input image processed by the encoder neural network and at each time step of a plurality steps:
• receive a glimpse captured by reading from the input image,
• receive a decoder hidden state vector of a decoder neural network for the preceding time step, and
• process the glimpse, the decoder hidden state vector, and an encoder hidden state vector of the encoder neural network from the preceding time step to generate an encoder hidden state vector for the time step;
• a decoder neural network, wherein the decoder neural network is a recurrent neural network that is configured to, for each of the plurality of time steps:
• receive a decoder input for the time step, and
• process the decoder hidden state vector for the preceding time step and the decoder input to generate a decoder hidden state vector for the time step;
• and
• a subsystem, wherein the subsystem is configured to generate a final output image by repeatedly updating a neural network output at each of a plurality of time steps based on an input image to generate a final neural network output and generating the final output image from the final neural network output, the updating comprising, for each of the time steps:
• reading the glimpse from the input image using the decoder hidden state vector for the preceding time step;
• providing the glimpse as input to the encoder neural network;
• generating the encoder hidden state vector at the time step as output from the encoder neural network;
• generating the decoder input for the decoder neural network from the encoder hidden state vector at the time step;
• providing the decoder input as input to the decoder neural network for the time step;
• generating a neural network output update for the time step from the decoder hidden state vector for the time step;
• and
• combining the neural network output update for the time step with a current neural network output to generate an updated neural network output.

A neural network system implemented by one or more computers, the neural network system comprising:

• a decoder neural network, wherein the decoder neural network is a recurrent neural network that is configured to, for each of the plurality of time steps:
• receive a decoder input for the time step, and
• process the decoder hidden state vector for the preceding time step and the decoder input to generate a decoder hidden state vector for the time step;
• and
• a subsystem, wherein the subsystem is configured to generate a final output image by repeatedly updating a neural network output at each of a plurality of time steps based on an input image to generate a final neural network output and generating the final output image from the final neural network output, the updating comprising, for each of the time steps:
• generating the decoder input for the decoder neural network;
• providing the decoder input as input to the decoder neural network for the time step;
• generating a neural network output update for the time step from the decoder hidden state vector for the time step;
• and
• combining the neural network output update for the time step with a current neural network output to generate an updated neural network output.