Video prediction using spatially displaced convolution

A system, comprising:

• a memory storing a sequence of video frames and at least one optical flow, each optical flow in the at least one optical flow corresponding to a pair of video frames in the sequence of video frames;
• at least one parallel processing unit coupled to the memory and configured to implement, at least in part:
• a first neural network configured to process the sequence of video frames and the at least one optical flow to generate a set of parameters that include a displacement vector and at least one convolution kernel for each pixel of a plurality of pixels in a predicted video frame, and
• a spatially-displaced convolution (SDC) module configured to generate the predicted video frame based on the set of parameters.

A non-transitory computer-readable media storing computer instructions for performing video prediction that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• receiving an input that includes the sequence of video frames and at least one optical flow, wherein each optical flow maps pixels from a particular video frame to motion vectors that identify corresponding pixels in a corresponding video frame in the sequence of video frames;
• processing the input by layers of a first neural network to generate a set of parameters that include a displacement vector and at least one convolution kernel for each pixel of a plurality of pixels in the predicted video frame;
• and
• generating the predicted video frame based on the set of parameters.