Kernel-Based Frame Interpolation for Spatio-Temporally Adaptive Rendering

21:

• A system comprising:
• a hardware processor;
• and
• a system memory storing a software code;
• the hardware processor configured to execute the software code to:
• receive a first rendered image and a second rendered image;
• provide, using a first machine learning (ML) model, a plurality of features of each of the first rendered image and the second rendered image;
• interpolate, based on the plurality of features, an intermediate image between the first rendered image and the second rendered image to provide an interpolated image;
• determine, using a third ML model and based on the plurality of features, a rendering mask for producing one or more rendered sub-portions of the intermediate image;
• partially render, using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions;
• and
• blend the interpolated image and the partially rendered image to provide a composite image.

26:

• The system of claim 21, wherein the hardware processor is further configured to execute the software code to:
• determine a plurality of kernels for interpolating the intermediate image, wherein the plurality of kernels are determined so as to ensure that the interpolated image comprises a linear combination of the first and second rendered images;
• and
• wherein interpolating the intermediate image to provide the interpolated image comprises using the plurality of kernels.

28:

• The system of claim 21, wherein the hardware processor is further configured to execute the software code to:
• compute a first optical flow from the first rendered image to the intermediate image;
• compute a second optical flow from the second rendered image to the intermediate image;
• splat, using the first optical flow, one or more of the plurality of features of the first rendered image to provide one or more of a first plurality of splatted features;
• and
• splat, using the second optical flow, one or more of the plurality of features of the second rendered image to provide one or more of a second plurality of splatted features;
• wherein interpolating the intermediate image to provide the interpolated image uses the first plurality of splatted features and the second plurality of splatted features.

31:

• A method comprising:
• receiving a first rendered image and a second rendered image;
• providing, using a first machine learning (ML) model, a plurality of features of each of the first rendered image and the second rendered image;
• interpolating, based on the plurality of features, an intermediate image between the first rendered image and the second rendered image to provide an interpolated image;
• determining, using a third ML model and based on the plurality of features, a rendering mask for producing one or more rendered sub-portions of the intermediate image;
• partially rendering, using the rendering mask, the intermediate image to provide a partially rendered image including the one or more rendered sub-portions;
• and
• blending the interpolated image and the partially rendered image to provide a composite image.

36:

• The method of claim 31, further comprising:
• determining a plurality of kernels for interpolating the intermediate image, wherein the plurality of kernels are determined so as to ensure that the interpolated image comprises a linear combination of the first and second rendered images;
• and
• wherein interpolating the intermediate image to provide the interpolated image comprises using the plurality of kernels.

38:

• The method of claim 31, further comprising:
• computing a first optical flow from the first rendered image to the intermediate image;
• computing a second optical flow from the second rendered image to the intermediate image;
• splatting, using the first optical flow, one or more of the plurality of features of the first rendered image to provide one or more of a first plurality of splatted features;
• and
• splatting, using the second optical flow, one or more of the plurality of features of the second rendered image to provide one or more of a second plurality of splatted features;
• wherein interpolating the intermediate image to provide the interpolated image uses the first plurality of splatted features and the second plurality of splatted features.