Differentiable real-time radiance field rendering for large scale view synthesis

A device, comprising:

• a non-transitory memory;
• and
• a processor configured to execute instructions stored in the non-transitory memory to:
• obtain images of an environment that are captured by one or more image capture devices, wherein at least one of intrinsic parameters or extrinsic parameters is different between the two or more image capture devices;
• determine the intrinsic parameters and the extrinsic parameters of the one or more image capture devices that are associated with each of the images, wherein the extrinsic parameters include spatial location coordinates and a viewing direction with respect to a reference image;
• determine neural graphics primitives of the environment based upon the images of the environment, wherein the neural graphics primitives comprise learned representations containing information about geometric and appearance features of the environment extracted from the images using a neural network trained to optimize an underlying continuous three-dimensional representation function using a sparse set of input views of the environment;
• create, a differentiable radiance field associated with the environment based upon the intrinsic parameters, the extrinsic parameters, and the neural graphics primitives,
• wherein the differentiable radiance field is a neural representation that models three-dimensional scene properties of the environment, including color and density, as continuous functions optimized via differentiable optimization training to enable rendering from novel viewpoints;
• and
• generate, based upon the differentiable radiance field, a three-dimensional representation of the environment, wherein the three-dimensional representation contains one or more portions of the environment not captured in the images.

A non-transitory computer-readable medium that stores computer-executable instructions that, when executed by a processor, perform operations comprising:

• obtaining images of an environment that are captured by one or more image capture devices, wherein at least one of intrinsic parameters or extrinsic parameters is different between the two or more image capture devices;
• determining the intrinsic parameters and the extrinsic parameters of the one or more image capture devices that are associated with each of the images, wherein the extrinsic parameters include spatial location coordinates and a viewing direction with respect to a reference image;
• determining neural graphics primitives of the environment based upon the images of the environment, wherein the neural graphics primitives comprise learned representations containing information about geometric and appearance features of the environment extracted from the images using a neural network trained to optimize an underlying continuous three-dimensional representation function using a sparse set of input views of the environment;
• creating a differentiable radiance field associated with the environment based upon the intrinsic parameters, the extrinsic parameters, and the neural graphics primitives,
• wherein the differentiable radiance field is a neural representation that models three-dimensional scene properties of the environment, including color and density, as continuous functions optimized via differentiable optimization training to enable rendering from novel viewpoints;
• and
• generating, based upon the differentiable radiance field, a three-dimensional representation of the environment, wherein the three-dimensional representation contains one or more portions of the environment not captured in the images.