Recurrent unit for generating or processing a sequence of images

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

• a recurrent unit arranged, at each of a series of time steps, to receive a corresponding input vector and to generate an output having at least one respective value for each of a two -dimensional array of pixels,
• the recurrent unit being configured at each of the series of time steps except a first time step:
• to receive the output of the recurrent unit at a preceding time step,
• to apply to the output of the recurrent unit at the preceding time step at least one convolution dependent on both the input vector at the time step and the output of the recurrent unit at the preceding time step, to generate a warped dataset which has at least one component for each pixel of the array, wherein the at least one component of the warped dataset for each pixel of the array is generated by convolving the output of the recurrent unit at the preceding time step with a respective kernel that is determined based on an output of a neural network that receives the input vector and the output of the recurrent unit at the preceding time step, and
• to generate the output at the time step based on the warped dataset and the input vector.

A computer-implemented method, comprising:

• receiving, by a recurrent unit, at each of a series of time steps, a corresponding input vector to generate an output having at least one respective value for each of a two-dimensional array of pixels, wherein, at each of the series of time steps except a first time step, the recurrent unit performs operations comprising:
• receiving the output of the recurrent unit at a preceding time step;
• applying, to the output of the recurrent unit at the preceding time step, at least one convolution dependent on both the input vector at the time step and the output of the recurrent unit at the preceding time step, to generate a warped dataset which has at least one component for each pixel of the array, wherein the at least one component of the warped dataset for each pixel of the array is generated by convolving the output of the recurrent unit at the preceding time step with a respective kernel that is determined based on an output of a neural network that receives the input vector and the output of the recurrent unit at the preceding time step, and
• generating the output at the time step based on the warped dataset and the input vector.