Novel view synthesis from sparse volume data structure

A computer-implemented method for rendering an image of a subject in a 3-dimensional space, the method comprising:

• processing rays from a view point through pixels of the image into the 3-dimensional space, the 3-dimensional space being represented by a sparse volume data structure and a residual neural network;
• sampling voxels along the rays and accumulating colors for the voxels according to color values stored within the sparse volume data structure;
• providing a location and viewing angle of the view point to the residual neural network to obtain residual colors for the pixels of the image;
• and
• rendering the image by combining the accumulated colors and the residual colors for each pixel of the image.

A non-transient computer-readable storage medium comprising instructions being executable by one or more processors, that when executed by the one or more processors, cause the one or more processors to:

• provide a plurality of inputs to a neural radiance field (NeRF) model that represents a 3-dimensional space having a subject, wherein each input of the plurality of inputs includes a location and a view direction and corresponds to respective colors of voxels that represent the 3-dimensional space;
• perform a spectral analysis on a plurality of outputs of the NeRF model based on the plurality of inputs, wherein the plurality of outputs include the respective colors of the voxels;
• extract frequency components of the spectral analysis that represent colors for at least some of the voxels;
• and
• generate a sparse volume data structure that represents the 3-dimensional space and the respective colors for the at least some of the voxels.