# US 11,908,071 B2 · Systems and methods for reconstructing body shape and pose > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,071 B2 - Document ID: US11908071B2 - Status: Granted - Kind: B2 - Application number: 17/495,960 - Filed: 2021-10-07 - Published: 2023-04-13 - Granted: 2024-02-20 - Earliest claimed: 2021-10-07 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Cristian Sminchisescu; Mihai Zanfir; Andrei Zanfir; Eduard Gabriel Bazavan; William Tafel Freeman; Rahul Sukthankar - Web page: https://patentgrep.com/patent/US11908071B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 20/00: Machine learning - G06T 7/73: using feature-based methods - G06T 12/00: Tomographic reconstruction from projections - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30204: Marker - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present disclosure is generally directed to reconstructing representations of bodies from images. An example method of the present disclosure includes inputting, into a machine-learned reconstruction model, input data descriptive of an image depicting a body; predicting, using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and outputting, using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations. In the example method, one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between relaxed-constraint representations generated from a prior set of surface marker locations predicted according to the one or more parameters and parametric representations generated from the prior set using kinematic constraints associated with the body. ## Claims ### Claim 1 A computer-implemented method for reconstructing representations of bodies from images, the method comprising: inputting, by one or more computing devices into a machine-learned reconstruction model, input data descriptive of an image depicting a body; predicting, by the one or more computing devices and using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and outputting, by the one or more computing devices and using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations; wherein one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between (i) a relaxed-constraint representation generated from a prior set of surface marker locations predicted according to the one or more parameters and (ii) a parametric representation generated from the prior set using kinematic constraints associated with the body. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the body is a human body and the kinematic constraints correspond to anthropometric constraints. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the output representation is a relaxed-constraint representation. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the marker prediction component comprises one or more encoder layers. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the one or more encoder layers respectively comprise self-attention models. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein predicting the set of surface marker locations comprises: encoding, by the one or more computing devices and using the one or more encoder layers, a surface marker embedding along with the input data; and updating, by the one or more computing devices, the set of surface marker locations based at least in part on the encoded surface marker embedding. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein an output of each of the one or more encoder layers is used to iteratively refine the set of surface marker locations, the output corresponding to the surface marker embedding. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the one or more encoder layers comprise a plurality of encoder layers that share one or more machine-learned weights. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising: transforming, by the one or more computing devices and using a capture model, the input data; and wherein the output representation is obtained in a capture space corresponding to the capture model. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the capture model is based at least in part on a perspective model. ### Claim 11 A system for reconstructing representations of bodies from images, comprising: one or more processors; and one or more memory devices storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: inputting, into a machine-learned reconstruction model, input data descriptive of an image depicting a body; predicting, using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and outputting, using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations; wherein one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between (i) a relaxed-constraint representation generated from a prior set of surface marker locations predicted according to the one or more parameters and (ii) a parametric representation generated from the prior set using kinematic constraints associated with the body. ### Claim 12 (depends on 11) The system of claim 11, wherein the body is a human body and the kinematic constraints correspond to anthropometric constraints. ### Claim 13 (depends on 11) The system of claim 11, wherein the output representation is a relaxed- constraint representation. ### Claim 14 (depends on 11) The system of claim 11, wherein the marker prediction component comprises one or more encoder layers. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more encoder layers respectively comprise self-attention models. ### Claim 16 (depends on 15) The system of claim 15, wherein predicting the set of surface marker locations comprises: encoding, using the one or more encoder layers, a surface marker embedding along with the input data; and updating the set of surface marker locations based at least in part on the encoded surface marker embedding. ### Claim 17 (depends on 16) The system of claim 16, wherein an output of each of the one or more encoder layers is used to iteratively refine the set of surface marker locations, the output corresponding to the surface marker embedding. ### Claim 18 (depends on 17) The system of claim 17, wherein the one or more encoder layers comprise a plurality of encoder layers that share one or more machine-learned weights. ### Claim 19 A system for reconstructing representations of bodies from images, comprising: one or more processors; and one or more memory devices storing computer-readable instructions that, when implemented, cause the one or more processors to perform operations, the operations comprising: inputting, into a machine-learned marker prediction model, input data descriptive of an image depicting a body; predicting, using the marker prediction model, a set of surface marker locations on the body; outputting, using a machine-learned marker poser model, a parametric representation of the body that corresponds to the set of surface marker locations; and updating one or more parameters of the marker prediction model based at least in part on a consistency loss corresponding to a distance between the parametric representation and a relaxed-constraint representation associated with the predicted set of surface marker locations. ### Claim 20 (depends on 19) The system of claim 19, wherein the output representation is a relaxed-constraint representation.