US 10,896,356 B2 · Efficient CNN-based solution for video frame interpolation

Abstract

A system of convolutional neural networks (CNNs) that synthesize middle non-existing frames from pairs of input frames includes a coarse CNN that receives a pair of images acquired at consecutive points of time, a registration module, a refinement CNN, an adder, and a motion-compensated frame interpolation (MC-FI) module. The coarse CNN outputs from the pair of images a previous feature map, a next feature map, a coarse interpolated motion vector field (IMVF) and an occlusion map, the registration module uses the coarse IMVF to warp the previous and next feature maps to be aligned with pixel locations of the IMVF frame, and outputs registered previous and next feature maps, the refinement CNN uses the registered previous and next feature maps to correct the coarse IMVF, and the adder sums the coarse IMVF with the correction and outputs a final IMVF.

Claims20

Dependent claims

A system that uses convolutional neural networks (CNNs) to synthesize middle non-existing frames from pairs of input frames in a given video, comprising:

a coarse convolutional neural network (CNN) that receives a pair of images acquired at consecutive points of time, wherein the pair of images includes a previous image and a next image;
a registration module connected to the coarse CNN;
a refinement CNN connected to the registration module and the coarse CNN;
an adder connected to the refinement CNN and the coarse CNN;
and
a motion-compensated frame interpolation (MC-FI) module connected to the adder and the coarse CNN,
wherein the coarse CNN outputs a previous feature map and a next feature map from the previous image and the next image, a coarse interpolated motion vector field (IMVF) and an occlusion map from the pair of images, the registration module uses the coarse IMVF to warp the previous and next feature maps to be aligned with pixel locations of the IMVF frame, and outputs a registered previous feature map and a registered next feature map, the refinement CNN uses the registered previous feature map and a registered next feature map to correct the coarse IMVF, and the adder sums the coarse IMVF with the correction to the IMVF and outputs a final IMVF.

4 more elements

A method of using convolutional neural networks (CNNs) to synthesize middle non-existing frames from pairs of input frames in a given video, comprising the steps of:

receiving a pyramid representation of a pair of consecutive input frames, wherein the pair of consecutive input frames includes a previous image and a next image, wherein the pyramid representation includes a plurality of pairs of input frames, each at a different spatial resolution level;
generating a pair of feature maps from each resolution level of the pyramid representation and estimating a coarse interpolated motion vector field (IMVF) and an occlusion map from each pair of feature maps;
registering pairs of feature maps at the same resolution level according to the coarse IMVF and the occlusion map by warping each feature map of the pair of feature maps to be aligned with pixel locations of the coarse IMVF and outputting a registered previous feature map and a registered next feature map;
correcting the coarse IMVF using the registered previous feature map and the registered next feature map to generate a correction to the IMVF;
adding the correction to the IMVF to the coarse IMVF to generate a refined IMVF;
and
producing a synthesized middle frame from the pair of consecutive input frames, the refined IMVF and the occlusion map.

4 more elements

A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executed by the computer to perform the method steps for using convolutional neural networks (CNNs) to synthesize middle non-existing frames from pairs of input frames in a given video, comprising the steps of:

receiving a pyramid representation of a pair of consecutive input frames, wherein the pair of consecutive input frames includes a previous image and a next image, wherein the pyramid representation includes a plurality of pairs of input frames, each at a different spatial resolution level;
generating a pair of feature maps from each resolution level of the pyramid representation and estimating a coarse interpolated motion vector field (IMVF) and an occlusion map from each pair of feature maps;
registering pairs of feature maps at the same resolution level according to the coarse IMVF and the occlusion map by warping each feature map of the pair of feature maps to be aligned with pixel locations of the coarse IMVF and outputting a registered previous feature map and a registered next feature map;
correcting the coarse IMVF using the registered previous feature map and the registered next feature map to generate a correction to the IMVF;
adding the correction to the IMVF to the coarse IMVF to generate a refined IMVF;
and
producing a synthesized middle frame from the pair of consecutive input frames, the refined IMVF and the occlusion map.

4 more elements

Dependencies are inferred from the claim text.

Efficient CNN-based solution for video frame interpolation

Life of this patent

Efficient CNN-based solution for video frame interpolation

Life of this patent

Abstract

Claims20