Context-aware synthesis for video frame interpolation

A computer system comprising:

• processor circuitry coupled with memory circuitry, the memory circuitry is arranged to store program code of flow estimation logic, context extraction logic, warping logic, and frame synthesis neural network (FSNN) logic, and the processor circuitry is arranged to:
• operate the flow estimation logic to estimate a bidirectional optical flow between at least two input frames;
• operate the context extraction logic to extract context maps based on the estimated bidirectional flow;
• operate the warp logic to pre-warp the at least two input frames and corresponding context maps of the input frames;
• operate the warp logic to feed the pre-warped frames and the corresponding context maps into an FSNN of the FSNN logic;
• and
• operate the FSNN logic to generate an output frame at a desired temporal position based on the pre-warped frames.

A computer-implemented method comprising:

• estimating a bidirectional optical flow between at least two input frames;
• extracting context maps based on the estimated bidirectional flow, wherein the context maps comprise per-pixel context information from the input frames;
• warping the at least two input frames and corresponding context maps of the input frames, wherein the pre-warping comprises using the bidirectional optical flow as a guide for the pre-warping;
• feeding the pre-warped frames and the corresponding context maps into a frame synthesis neural network (FSNN);
• and
• operating the FSNN to generate an output frame at a desired temporal position based on the pre-warped frames.

One or more non-transitory computer-readable media (NTCRM) comprising instructions, wherein execution of the instructions by one or more processors of a computing system is operable to cause the computing system to:

• estimate a bidirectional optical flow between at least two input frames;
• extract context maps based on the estimated bidirectional flow;
• warp the at least two input frames and corresponding context maps of the input frames;
• feed the warped frames and the corresponding context maps into a frame synthesis neural network (FSNN);
• and
• operate the FSNN to generate an output frame at a desired temporal position based on the warped frames.