Animatable garment extraction through volumetric reconstruction

A system comprising:

• at least one processor;
• and
• at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
• accessing a monocular image depicting a person wearing a fashion item;
• generating a three-dimensional (3D) mesh representing the person in a canonical space;
• analyzing the monocular image to predict and simultaneously output both a volumetric reconstruction tensor of the monocular image and a pose of the person using a first machine learning model, the volumetric reconstruction tensor comprising estimated pixel values of 3D points of portions of the monocular image, the first machine learning model trained by performing training operations comprising:
• predicting both an individual volumetric reconstruction tensor of the monocular image and an individual pose of a first object by applying the first machine learning model to a first training image;
• identifying an individual portion of the individual pose of the first object that corresponds to an individual point in the canonical space associated with an individual set of position-encoding information;
• obtaining an individual point of the individual volumetric reconstruction tensor corresponding to the individual portion;
• classifying the individual point as being inside or outside of an individual canonical volume by applying a second machine learning model to the individual point of the volumetric reconstruction tensor together with the individual set of position-encoding information;
• and
• generating occupancy loss based on a deviation between a classification resulting from classifying the individual point and a corresponding portion of a first ground truth canonical space representation of the first object;
• modifying a pose of the 3D mesh to match the predicted pose of the person;
• extracting a portion of the 3D mesh corresponding to the fashion item;
• and
• generating an extended reality (XR) item using the extracted portion of the 3D mesh.

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• accessing a monocular image depicting a person wearing a fashion item;
• generating a three-dimensional (3D) mesh representing the person in a canonical space;
• analyzing the monocular image to predict and simultaneously output both a volumetric reconstruction tensor of the monocular image and a pose of the person using a first machine learning model, the volumetric reconstruction tensor comprising estimated pixel values of 3D points of portions of the monocular image, the first machine learning model trained by performing training operations comprising:
• predicting both an individual volumetric reconstruction tensor of the monocular image and an individual pose of a first object by applying the first machine learning model to a first training image;
• identifying an individual portion of the individual pose of the first object that corresponds to an individual point in the canonical space associated with an individual set of position-encoding information;
• obtaining an individual point of the individual volumetric reconstruction tensor corresponding to the individual portion;
• classifying the individual point as being inside or outside of an individual canonical volume by applying a second machine learning model to the individual point of the volumetric reconstruction tensor together with the individual set of position-encoding information;
• and
• generating occupancy loss based on a deviation between a classification resulting from classifying the individual point and a corresponding portion of a first ground truth canonical space representation of the first object;
• modifying a pose of the 3D mesh to match the predicted pose of the person;
• extracting a portion of the 3D mesh corresponding to the fashion item;
• and
• generating an extended reality (XR) item using the extracted portion of the 3D mesh.