Deep loop filter by temporal deformable convolution
AssigneeTencent America LLC
InventorsWei Jiang, Wei Wang, Zeqiang Li, Shan Liu
Life of this patent
Estimating the term…
Abstract
A method, apparatus and storage medium for performing video coding are provided. The method includes obtaining a plurality of image frames in a video sequence; determining a feature map for each of the plurality of image frames and determining an offset map based on the feature map; determining an aligned feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and generating a plurality of aligned frames based on the aligned feature map.
Claims18
A method of performing video coding using one or more neural networks with a loop filter, the method comprising:
- obtaining a plurality of image frames in a video sequence;
- determining a feature map for each of the plurality of image frames;
- selecting a reference frame among the plurality of image frames, the reference frame being a frame to which other frames in the plurality of image frames are to be aligned;
- concatenating a reference feature map of the reference frame and a feature map of each of the other frames in the plurality of image frames and passing a concatenated feature map through an offset generation Deep Neural Network (DNN), to generate an offset map;
- determining an aligned feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map;
- and
- generating a plurality of aligned frames based on the aligned feature map.
An apparatus comprising:
- at least one memory storing computer program code;
- and
- at least one processor configured to access the at least one memory and operate as instructed by the computer program code, the computer program code comprising:
- obtaining code configured to cause the at least one processor to obtain a plurality of image frames in a video sequence;
- determining code configured to cause the at least one processor to:
- determine a feature map for each of the plurality of image frames;
- select a reference frame among the plurality of image frames, the reference frame being a frame to which other frames in the plurality of image frames are to be aligned;
- concatenate a reference feature map of the reference frame and a feature map of each of the other frames in the plurality of image frames and passing a concatenated feature map through an offset generation Deep Neural Network (DNN), to generate an offset map;
- and
- determine an aligned feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map;
- and
- generating code configured to cause the at least one processor to generate a plurality of aligned frames based on the aligned feature map.
A non-transitory computer-readable storage medium storing computer program code, the computer program code, when executed by at least one processor, the at least one processor is configured to:
- obtain a plurality of image frames in a video sequence;
- determine a feature map for each of the plurality of image frames;
- selecting a reference frame among the plurality of image frames, the reference frame being a frame to which other frames in the plurality of image frames are to be aligned;
- concatenating a reference feature map of the reference frame and a feature map of each of the other frames in the plurality of image frames and passing a concatenated feature map through an offset generation Deep Neural Network (DNN), to generate an offset map;
- determine an aligned feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map;
- and
- generate a plurality of aligned frames based on the aligned feature map.
Dependencies are inferred from the claim text.