System and method for improving novel view synthesis using latent diffusion models in 3d gaussian splatting

An apparatus comprising a processor and a memory storing instructions that, when executed by the processor, cause the processor to:

• render an image using a three-dimensional (3D) Gaussian splatting process;
• process the rendered image with a pretrained latent diffusion model to estimate noise in a latent space;
• generate a diffusion loss based on a difference between the estimated noise and a sampled noise;
• periodically apply the diffusion loss to update parameters of the 3D Gaussian splatting process;
• and
• generate a novel view synthesis image based on the updated parameters.