View Synthesis Robust To Unconstrained Image Data

A computer-implemented method for view synthesis with user-specifiable characteristics, the method comprising:

• obtaining, by a computing system comprising one or more computing devices, a desired position within three-dimensional space and a user-specified generative embedding, wherein the generative embedding encodes one or more visual characteristics of a synthetic image to be generated;
• processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation;
• processing, by the computing system, the latent representation and the generative embedding with a content portion of the machine-learned view synthesis model to generate a color;
• and
• performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of the synthesized image from the opacity and the color, wherein the synthesized image exhibits the one or more visual characteristics encoded by the generative embedding.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:

• obtaining, by the computing system, a position within three-dimensional space and a training embedding associated with an existing training image, wherein the training embedding encodes one or more visual characteristics of the existing training image;
• processing, by the computing system, data descriptive of the position with a base portion of a machine-learned view synthesis model to generate an opacity and a latent representation;
• processing, by the computing system, the latent representation and the training embedding with a content portion of the machine-learned view synthesis model to generate a color;
• and
• performing, by the computing system, volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the opacity and the color;
• evaluating a loss function that compares the synthetic pixel color with a ground truth pixel color for a training pixel including in the existing training image;
• and
• modifying one or more values of the training embedding based at least in part on the loss function.

A computing system to generate synthetic imagery of a scene, the computing system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store:
• a machine-learned view synthesis model that comprises a base portion and a static content portion that models static content within the scene, wherein the base portion and the static portion have been trained jointly with a transient content portion that models transient content within the scene;
• and
• instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
• obtaining a position within three-dimensional space;
• processing data descriptive of the position with the base portion of the machine-learned view synthesis model to generate a static opacity and a latent representation;
• processing the latent representation with the static content portion of the machine-learned view synthesis model to generate a static color;
• and
• performing volumetric rendering to generate a synthetic pixel color for a synthetic pixel of a synthesized image from the static opacity and the static color.