Self-supervised single-view 3D reconstruction via semantic consistency

A method of training one or more neural networks, comprising:

• training a first autoencoder, of the one or more neural networks, from a plurality of input sets, wherein an input set of the plurality of input sets comprises an input image of an object and an input image mask delineating, at least approximately, portions of the object in the input image, wherein the first autoencoder maps features of its input to a latent representation corresponding to a three-dimensional mesh template;
• generating a first mesh using the first autoencoder with a first input set of the plurality of input sets as a first input to the first autoencoder;
• generating a first output image mask, using the first autoencoder, that is based on the first mesh, the first output image mask including a semantic map that names parts of the object;
• comparing the first output image mask to a first input image mask of the first input set to determine first differences;
• adjusting a first encoder of the first autoencoder based on the first differences;
• training a second autoencoder, of the one or more neural networks, using arbitrary inputs, wherein the second autoencoder shares the latent representation and a decoder with the first autoencoder;
• generating a second mesh using the second autoencoder;
• comparing, using a discriminator, the first mesh and the second mesh for determining second differences;
• adjusting the latent representation based on the second differences and/or the first differences;
• and
• adjusting the decoder based on the second differences and/or the first differences.

A system comprising:

• a processor comprising one or more arithmetic logic units (ALUs);
• one or more memories to store a representation of a three-dimensional mesh template comprising a plurality of vertices, representable as locations in a three-dimensional space, and a plurality of faces each defined by three or more vertices of the plurality of vertices;
• an input for receiving an instance image of an object and an object mask delineating, at least approximately, portions of the object in the instance image, the object being an instance of an object category;
• and
• wherein the processor is programmed to:
• process the instance image and the object mask using a neural network trained on training images of objects of the object category;
• derive, based on the processing, an estimated three-dimensional mesh for the object in the instance image;
• generating a semantic map corresponding to semantic parts of the objects in the object category;
• derive, based on the processing, an estimated texture of the object in the instance image;
• derive, based on the processing, an estimated camera pose, wherein the estimated camera pose corresponds to a position of the estimated three-dimensional mesh relative to the object in the instance image;
• and
• generate a reconstructed image from the estimated three-dimensional mesh, the estimated texture, and a desired camera pose.

A system, comprising memory to store instructions that, as a result of execution by one or more processors, causes the system to:

• obtain a representation of a three-dimensional mesh template comprising a plurality of vertices, representable as locations in a three-dimensional space, and a plurality of faces each defined by three or more vertices of the plurality of vertices;
• receive an instance image of an object;
• receive an object mask delineating, at least approximately, portions of the object in the instance image, the object being an instance of an object category;
• process the instance image and the object mask using a neural network trained on training images of objects of the object category;
• derive an estimated three-dimensional mesh for the object in the instance image, at least in part using the three-dimensional mesh template;
• generate a semantic map corresponding to semantic parts of the objects in the object category;
• derive an estimated texture of the object in the instance image, at least in part from the instance image and the estimated three-dimensional mesh;
• derive an estimated camera pose, wherein the estimated camera pose corresponds to a position of the estimated three-dimensional mesh relative to the object in the instance image;
• generate a reconstruction image from the estimated three-dimensional mesh, the estimated texture, and the estimated camera pose;
• compare the reconstruction image and the instance image to determine a loss function value of a loss function;
• and
• update one or more of the estimated three-dimensional mesh, the estimated texture, or the estimated camera pose based on the loss function value.