Unifying densepose and 3D body mesh reconstruction

A system comprising:

• at least one processor;
• and
• at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
• receiving an image that includes a depiction of a real-world object;
• predicting a tensor of two-dimensional (2D) heatmaps by applying a first machine learning model to a portion of the image, each 2D heatmap of the tensor of 2D heatmaps exclusively representing possible positions for a corresponding single vertex of an individual triangle of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object, the first machine learning model trained to predict a part of the 3D mesh that is not visible within the image including at least one of occluded vertices, self-occluded vertices, or truncated vertices, a first heatmap of the tensor of heatmaps representing a first group of possible coordinates exclusively for a single first vertex of a first triangle of the plurality of triangles, a second heatmap of the tensor of heatmaps representing a second group of possible coordinates exclusively for a single second vertex of the same first triangle as the first heatmap;
• selecting a subset of the tensor of heatmaps from the tensor of heatmaps, the subset of the tensor of heatmaps comprising a first coordinate of the first group of possible coordinates and a second coordinate of the second group of possible coordinates;
• and
• generating the 3D mesh based on the selected subset of the tensor of heatmaps, the 3D mesh being combined with an inpainted mesh to form a reconstructed 3D mesh.

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• receiving an image that includes a depiction of a real-world object;
• predicting a tensor of two-dimensional (2D) heatmaps by applying a first machine learning model to a portion of the image, each 2D heatmap of the tensor of 2D heatmaps exclusively representing possible positions for a corresponding single vertex of an individual triangle of a plurality of triangles of a three-dimensional (3D) mesh corresponding to the real-world object, the first machine learning model trained to predict a part of the 3D mesh that is not visible within the image including at least one of occluded vertices, self-occluded vertices, or truncated vertices, a first heatmap of the tensor of heatmaps representing a first group of possible coordinates exclusively for a single first vertex of a first triangle of the plurality of triangles, a second heatmap of the tensor of heatmaps representing a second group of possible coordinates exclusively for a single second vertex of the same first triangle as the first heatmap;
• selecting a subset of the tensor of heatmaps from the tensor of heatmaps, the subset of the tensor of heatmaps comprising a first coordinate of the first group of possible coordinates and a second coordinate of the second group of possible coordinates;
• and
• generating the 3D mesh based on the selected subset of the tensor of heatmaps, the 3D mesh being combined with an inpainted mesh to form a reconstructed 3D mesh.