Point-based neural radiance field for three dimensional scene representation

A system comprising:

• a memory component configured to store a plurality of input two-dimensional (2D) images corresponding to a plurality of views of an object;
• and
• a processing device coupled to the memory component to perform operations comprising:
• receiving a request to generate a three-dimensional (3D) scene that includes the object;
• extracting, from the input 2D images using a point cloud generation model, 2D image feature maps describing edges and corners of the input 2D images;
• generating, using the 2D image feature maps, a neural point cloud representing the 3D scene;
• determining, using a neural point volume rendering model applied to the neural point cloud, a color value for each pixel of an output image, each color value based on a shading point color value and a density value;
• and
• generating the output image of the 3D scene based on the neural point cloud and the color value for each pixel;
• and
• transmitting the output image of the 3D scene.

A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

• extracting, from a plurality of input two-dimensional (2D) images, 2D image feature maps describing edges and corners of the input 2D images;
• generating, using the 2D image feature maps, a neural point cloud representing a three-dimensional (3D) scene that includes an object represented in the input 2D images;
• determining, using the neural point cloud, a color value for each pixel of an output image, each color value based on a shading point color value and a density value;
• and
• generating the output image of the 3D scene based on the neural point cloud and the color value for each pixel;
• and
• storing the output image of the 3D scene.