US 2021/0279840 A1 · SYSTEMS AND METHODS FOR MULTI-FRAME VIDEO FRAME INTERPOLATION

Abstract

Systems and methods for multi-frame video frame interpolation. Higher-order motion modeling, such as cubic motion modeling, achieves predictions of intermediate optical flow between multiple interpolated frames, assisted by relaxation of the constraints imposed by the loss function used in initial optical flow estimation. A temporal pyramidal optical flow refinement module performs coarse-to-fine refinement of the optical flow maps used to generate the intermediate frames, focusing a proportionally greater amount of refinement attention to the optical flow maps for the high-error middle frames. A temporal pyramidal pixel refinement module performs coarse-to-fine refinement of the generated intermediate frames, focusing a proportionally greater amount of refinement attention to the high-error middle frames. A generative adversarial network (GAN) module calculates a loss function for training the neural networks used in the optical flow estimation module, temporal pyramidal optical flow refinement module, and/or temporal pyramidal pixel refinement module.

Claims20

Dependent claims

A method for multi-frame video frame interpolation, the method comprising:

obtaining a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time;
processing the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time;
processing the plurality of optical flow maps and the first refined intermediate optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time;
generating a first new intermediate frame based on the first refined optical flow map;
and
generating a second new intermediate frame based on the second refined optical flow map.

3 more elements

A computing system comprising:

a processor;
and
a memory storing processor-executable instructions thereon, wherein the instructions, when executed by the processor, cause the computing system to:
obtain a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time;
processing the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time;
processing the plurality of optical flow maps and the first refined optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time;
generate a first new intermediate frame based on the first refined optical flow map;
and
generate a second new intermediate frame based on the second refined optical flow map.

6 more elements

A multi-frame video frame interpolation system, comprising:

an optical flow refinement module configured to:
obtain a plurality of optical flow maps corresponding to a plurality of times between a beginning time and an ending time;
process the plurality of optical flow maps to generate a first refined optical flow map corresponding to a first intermediate time proximal to the beginning time or ending time;
process the plurality of optical flow maps and the first refined intermediate optical flow map to generate a second refined optical flow map corresponding to a second intermediate time distal from both the beginning time and ending time;
generate a first new intermediate frame based on the first refined optical flow map;
and
generate a second new intermediate frame based on the second refined optical flow map.

4 more elements

Dependencies are inferred from the claim text.

Systems and methods for multi-frame video frame interpolation

Life of this patent

Systems and methods for multi-frame video frame interpolation

Life of this patent

Abstract

Claims20