3d generation of diverse categories and scenes

A system for generating a three-dimensional (3D) object or scene from non-aligned generic camera priors, comprising:

• a 3D scene generator that produces a tri-plane representation for an input scene received in random latent code;
• a camera generator that obtains a camera posterior including posterior parameters representing color and density data from the random latent code and from generic camera priors without alignment assumptions of the generic camera priors;
• a volume renderer that volumetrically renders an image of the input scene from the color and density data to provide a scene having pixel colors and depth values from an arbitrary camera viewpoint;
• a depth adaptor that processes the depth values to generate an adapted depth map that bridges domains of rendered and estimated depth maps for the image of the input scene;
• and
• a discriminator that receives the adapted depth map, color data and external scene geometry information from an external dataset and selects a 3D representation of the input scene based on the color data, adapted depth map, and external scene geometry information.

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor cause the processor to generate a three-dimensional (3D) object or scene from non-aligned generic camera priors by performing operations comprising:

• producing a tri-plane representation for an input scene received in random latent code;
• obtaining a camera posterior including posterior parameters representing color and density data from the random latent code and from generic camera priors without alignment assumptions of the generic camera priors;
• volumetrically rendering an image of the input scene from the color and density data to provide a scene having pixel colors and depth values from an arbitrary camera viewpoint;
• processing the depth values to generate an adapted depth map that bridges domains of rendered and estimated depth maps for the image of the input scene;
• and
• selecting a 3D representation of the input scene based on the color data, adapted depth map, and external scene geometry information from an external dataset.