Feature pyramid warping for video frame interpolation

One or more non-transitory computer-readable media (NTCRM) comprising instructions of a frame interpolation neural network (FINN), wherein execution of the instructions by one or more processors is to cause the one or more processors to:

• obtain a first input frame and a second input frame of a video;
• estimate a forward optical flow and a backward optical flow from the first and second input frames, the forward optical flow indicating how pixels in the first input frame are to be changed to produce the second input frame during a time period starting from the first input frame and ending at the second input frame, and the backward optical flow indicating how pixels in the second input frame are to be changed to produce the first input frame during a time period starting from the first input frame and ending at the second input frame;
• extract a first feature pyramid from the first input frame and a second feature pyramid from the second input frame, the first feature pyramid including a first set of features extracted from the first input frame at each resolution of a plurality of resolutions, and the second feature pyramid including a second set of features extracted from the second input frame at each resolution of the plurality of resolutions;
• warp the first feature pyramid toward the second feature pyramid using the forward optical flow;
• warp the second feature pyramid toward the first feature pyramid using the backward optical flow;
• and
• generate an output frame at a temporal position between the first and second input frames based on the warped first and second feature pyramids.

A computing system comprising:

• processor circuitry coupled with memory circuitry, the memory circuitry arranged to store program code of a frame interpolation neural network (FINN), the FINN comprising an optical flow estimator (OFE), a feature pyramid extractor (FPE), a forward warping engine (FWE), and a frame synthesis neural network (FSN), and
• the processor circuitry is arranged to operate the OFE to estimate a forward optical flow and a backward optical flow from first and second input frames of a video to be interpolated, the forward optical flow indicating how pixels in the first input frame are to be changed to produce the second input frame during a time period starting from the first input frame and ending at the second input frame, and the backward optical flow indicating how pixels in the second input frame are to be changed to produce the first input frame during a time period starting from the first input frame and ending at the second input frame;
• the processor circuitry is arranged to operate the FPE to extract a first feature pyramid from the first input frame and a second feature pyramid from the second input frame, the first feature pyramid including a first set of features extracted from the first input frame at each resolution of a plurality of resolutions, and the second feature pyramid including a second set of features extracted from the second input frame at each resolution of the plurality of resolutions;
• the processor circuitry is arranged to operate the FWE to warp the first feature pyramid toward the second feature pyramid using the forward optical flow, and warp the second feature pyramid toward the first feature pyramid using the backward optical flow;
• the processor circuitry is arranged to operate the FSN to generate an output frame at a desired temporal position between the first and second input frames based on the warped first and second feature pyramids, wherein the output frame includes pixels of the first and the second input frames shifted from the first and second input frames, respectively, to replicate motion to take place from the first input frame to the target temporal location and from the target temporal location to the second input frame.