Frame interpolation with multi-scale deep loss functions and generative adversarial networks

A system comprising:

• a processor;
• and
• a memory including code segments that when executed by the processor cause the processor to:
• select two or more frames from a plurality of frames of a video;
• downscale the two or more frames,
• estimate flow data based on the downscaled two or more frames using a first neural network, wherein the neural network is a trained optical flow model that progressively applies and refines flow fields,
• upscale the estimated flow data,
• generate, using a second neural network that is trained to estimate optical flow fields, refined flow data based on the upscaled estimated flow data and the downscaled two or more frames, the upscaled estimated flow data and the downscaled two or more frames having a same scaling factor,
• upscale the refined flow data, and
• synthesize an image based on the upscaled refined flow data and the two or more frames;
• and
• insert the image between the two or more frames as an interpolated frame in the video.

A non-transitory computer readable storage medium including instructions that when executed by at least one processor, cause the at least one processor to:

• select two or more frames from a plurality of frames of a video;
• downscale the two or more frames;
• estimate flow data based on the downscaled two or more frames using a first neural network, wherein the neural network is a trained optical flow model that progressively applies and refines flow fields;
• upscale the estimated flow data;
• generate, using a second neural network that is trained to estimate optical flow fields, refined flow data based on the upscaled estimated flow data and the downscaled two or more frames, the upscaled estimated flow data and the downscaled two or more frames having a same scaling factor;
• upscale the refined flow data;
• synthesize an image based on the upscaled refined flow data and the two or more frames;
• and
• insert the image as an interpolated frame into the plurality of frames of the video.