Geometry-aware three-dimensional synthesis in all angles

A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform one or more operations, comprising:

• pre-processing a first set of images and a second set of images by:
• scaling the first set of images to a predetermined size and aligning the first set of images at a center of a reference;
• centering a bounding box at the reference for the second set of images;
• and
• adjusting a scale and a translation of the center of the reference with constant offsets for the second set of images;
• and
• tuning a volume rendering for each image of the first and second sets of images by:
• associating each image with a latent code and synthesizing the latent code at a view of a camera pose;
• mapping a camera pose offset from the latent code and the camera pose;
• generating synthesized images based on the camera pose and the camera pose offset;
• and
• disentangling a foreground from a background of the synthesized images by:
• generating two-dimensional backgrounds for the first and second sets of images;
• determining a foreground mask during the volume rendering;
• and
• supervising, by a discriminator, a training of the foreground mask with an up-sampled image and a super-resolved image.

A system comprising:

• one or more computers;
• and
• one or more computer memory devices interoperably coupled with the one or more computers and having computer-readable storage media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising:
• pre-processing a first set of images and a second set of images by:
• scaling the first set of images to a predetermined size and aligning the first set of images at a center of a reference;
• centering a bounding box at the reference for the second set of images;
• and
• adjusting a scale and a translation of the center of the reference with constant offsets for the second set of images;
• and
• tuning a volume rendering for each image of the first and second sets of images by:
• associating each image with a latent code and synthesizing the latent code at a view of a camera pose;
• mapping a camera pose offset from the latent code and the camera pose;
• generating synthesized images based on the camera pose and the camera pose offset;
• and
• disentangling a foreground from a background of the synthesized images by:
• generating two-dimensional backgrounds for the first and second sets of images;
• determining a foreground mask during the volume rendering;
• and
• supervising, by a discriminator, a training of the foreground mask with an up-sampled image and a super-resolved image.