Three-dimensional object reconstruction from a video

A computer-implemented method of constructing a three-dimensional (3D) representation of an object, comprising:

• receiving, by a neural network model, a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting, by the neural network model, a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• predicting non-rigid motion deformations for the images;
• applying the non-rigid motion deformations to identity shapes predicted for the images to produce 3D shape representations of the object;
• evaluating a loss function based on rotated differences between the identity shapes and differences between the 3D shape representations;
• and
• updating parameters of the neural network model based on the loss function to reduce discontinuities in the 3D shape representations.

A computer-implemented method of constructing a three-dimensional (3D) representation of an object, comprising:

• receiving, by a neural network model, a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting, by the neural network model, a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• transferring the texture image predicted for the first image to a second 3D shape representation predicted for a second image of the images to produce a first 3D object;
• projecting the first 3D object according to a first camera pose associated with the first image to produce a first projected 3D object;
• transferring a second texture image predicted for the second image to the 3D shape representation predicted for the first image to produce a second 3D object;
• projecting the second 3D object according to a second camera pose associated with the second image to produce a second projected 3D object;
• and
• updating parameters of the neural network model to encourage consistency between the first projected 3D object and the second projected 3D object.

A computer-implemented method of constructing a three-dimensional (3D) representation of an object, comprising:

• receiving, by a neural network model, a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting, by the neural network model, a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• applying first non-rigid motion deformations predicted for the first image to a first identity shape predicted for a second image of the images to produce a first 3D shape representation;
• projecting the first 3D shape representation according to a first camera pose associated with the first image to produce a first projected 3D object;
• applying second non-rigid motion deformations predicted for the second image to a second identity shape predicted for the first image to produce a second 3D shape representation;
• projecting the second 3D shape representation according to a second camera pose associated with the second image to produce a second projected 3D object;
• and
• updating parameters of the neural network model to encourage consistency between the first projected 3D object and the second projected 3D object.

A computer-implemented method of constructing a three-dimensional (3D) representation of an object, comprising:

• receiving, by a neural network model, a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting, by the neural network model, a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• propagating a part map across the object in a number of the images to produce propagated part maps;
• mapping the propagated part maps into the texture space according to corresponding texture flows predicted for the number of images to produce part maps in the texture space;
• and
• aggregating the part maps to produce a video-level part map.

A system, comprising:

• a neural network model configured to construct a three-dimensional (3D) representation of an object by:
• receiving a video including images of the object captured from a camera pose;
• predicting a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• propagating a part map across the object in a number of the images to produce propagated part maps;
• mapping the propagated part maps into the texture space according to corresponding texture flows predicted for the number of images to produce part maps in the texture space;
• and
• aggregating the part maps to produce a video-level part map.

A non-transitory computer-readable media storing computer instructions for constructing a three-dimensional (3D) representation of an object that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• receiving, by a neural network model, a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases;
• predicting, by the neural network model, a texture flow for the first image;
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image;
• propagating a part map across the object in a number of the images to produce propagated part maps;
• mapping the propagated part maps into the texture space according to corresponding texture flows predicted for the number of images to produce part maps in the texture space;
• and
• aggregating the part maps to produce a video-level part map.

A system, comprising:

• a neural network model configured to construct a three-dimensional (3D) representation of an object by:
• extracting, by an encoder portion of the neural network model, features from a video including images of the object captured from a camera pose;
• predicting a 3D shape representation of the object for a first image of the images based on a set of learned shape bases and the features;
• predicting, by a texture decoder portion of the neural network model, a texture flow for the first image based on the features extracted from the first image;
• and
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image that is invariant to shape deformation of the object, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image.

A non-transitory computer-readable media storing computer instructions for constructing a three-dimensional (3D) representation of an object that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• extracting, by an encoder portion of a neural network model, features from a video including images of the object captured from a camera pose;
• predicting, by the neural network model, a 3D shape representation of the object for a first image of the images based on a set of learned shape bases and the features;
• predicting, by a texture decoder portion of the neural network model, a texture flow for the first image based on the features extracted from the first image;
• and
• mapping pixels from the first image to a texture space according to the texture flow to produce a texture image that is invariant to shape deformation of the object, wherein transfer of the texture image onto the 3D shape representation constructs a 3D object corresponding to the object in the first image.