View synthesis utilizing scene-level features and pixel-level features

One or more computer storage media comprising computer-executable instructions that when executed by computing device performs a method of training a diffusion model, the method comprising:

• receiving, by a computing device, a set of input images, each viewing direction of each of the set of input images, and a target viewing direction, the set of input images representing corresponding views of a scene from each viewing direction and the target viewing direction corresponding to a different view of the scene;
• generating, based on applying each of the set of input images to a first neural network, a plurality of feature maps corresponding to each of the set of input images, each feature map of the set of feature maps corresponding to pixels of each corresponding input image of the set of input images;
• generating, by using each of the set of feature maps as input to a global average pooling process, a scene feature representation;
• generating a plurality of projected pixel-point feature representations by applying the set of feature maps to a second neural network;
• generating, by a ray transformer taking the plurality of pixel feature representations the scene feature representation and the target viewing direction as input to the ray transformer, a plurality of ray feature representations corresponding to each point sampled along each ray in the target viewing direction of the different view;
• generating, from the plurality of ray feature representations, color values for each pixel of an output image representing the different view of the scene;
• and
• causing rendering of the output image using the color values as input to a rendering process.

A method of novel view synthesis, comprising:

• receiving, by a computing device, an input image, a viewing direction of the input image, and a target viewing direction, the input image representing a view of a scene from the viewing direction and the target viewing direction corresponding to a different view of the scene;
• generating, based on applying the input image to a first neural network, a feature map corresponding to pixels of the input image;
• generating, based on applying the feature map and viewing direction of the input image to a second neural network, a plurality of pixel feature representations corresponding to the pixels of the input image;
• generating, by using the feature map as input to a global average pooling process, an uncalibrated scene feature representation;
• generating, based on applying a rotation to the uncalibrated scene feature representation based on the viewing direction of the input image and the target viewing direction, a scene feature representation;
• generating, by a ray transformer taking the plurality of pixel feature representations, the scene feature representation and the target viewing direction as input, a plurality of ray feature representations corresponding to each point sampled along each ray in the target viewing direction of the different view;
• generating, from the plurality of ray feature representations, color values for each pixel of an output image representing the different view of the scene;
• and
• causing rendering of the output image using the color values as input to a rendering process.