Predicted frame generation by deformable convolution for video coding

A method of neural network training, comprising:

• inputting a set of reference frames to a predicted frame generation module to generate a predicted frame, the predicted frame generation module including neural networks having parameters that are to be optimized, the neural networks including a deformable convolutional deep neural network (DNN);
• determining a loss of a loss function, the loss function including:
• a compressive loss indicating a bitrate estimated based on a difference between the predicted frame and a ground-truth frame, and
• a reconstruction quality loss indicating a quality of the predicted frame with respect to the ground-truth frame;
• and
• performing a back-propagation based on the loss of the loss function to update the parameters of the neural networks in the predicted frame generation module.

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method of video coding, the method comprising:

• generating a first feature map using a feature extraction deep neural network (DNN) based on one or more previously reconstructed reference frames, the first feature map including one of a two-dimensional feature map, a three-dimensional feature map, or a four-dimensional (4D) feature tensor that is generated based on an input of a 4D tensor formed by the one or more previously reconstructed reference frames;
• performing a deformable convolution through a deformable convolutional DNN to generate a second feature map based on the first feature map;
• generating a predicted frame based on the second feature map;
• and
• reconstructing a current frame based on the predicted frame.