# US 12,223,577 B2 · Data-driven physics-based models with implicit actuations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,577 B2 - Document ID: US12223577B2 - Status: Granted - Kind: B2 - Application number: 18/159,651 - Filed: 2023-01-25 - Published: 2023-07-27 - Granted: 2025-02-11 - Earliest claimed: 2022-01-26 - Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Applicants: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich) - Inventors: Gaspard Zoss; Baran Gözcü; Barbara Solenthaler; Lingchen Yang; Byungsoo Kim - Web page: https://patentgrep.com/patent/US12223577B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 2210/36: Level of detail - G06T 2219/2021: Shape modification - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract One embodiment of the present invention sets forth a technique for generating actuation values based on a target shape such that the actuation values cause a simulator to output a simulated soft body that matches the target shape. The technique includes inputting a latent code that represents a target shape and a point on a geometric mesh into a first machine learning model. The technique further includes generating, via execution of the first machine learning model, one or more simulator control values that specify a deformation of the geometric mesh, where each of the simulator control values is based on the latent code and corresponds to the input point, and generating, via execution of the simulator, a simulated soft body based on the one or more simulator control values and the geometric mesh. The technique further includes causing the simulated soft body to be outputted to a computing device. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a latent code that represents a target shape; inputting the latent code and an input point on a geometric mesh associated with the target shape into a first machine learning model; generating, via execution of the first machine learning model, one or more simulator control values that specify a deformation of the geometric mesh, wherein each of the simulator control values is based on the latent code and corresponds to the input point on the geometric mesh; generating, via execution of a differentiable simulator, a simulated soft body based on the one or more simulator control values and the geometric mesh; and causing the simulated soft body to be outputted to a computing device, wherein the simulated soft body is used to perform one or more simulation operations. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein generating, via execution of the differentiable simulator, the simulated soft body comprises: generating, via execution of the differentiable simulator, an actuated geometric mesh by deforming the geometric mesh based on the one or more simulator control values; wherein the simulated soft body is generated based on the actuated geometric mesh. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: inputting the target shape and a specified resolution into an encoder machine learning model; and generating, via execution of the encoder machine learning model for the specified resolution, the latent code that represents the target shape. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the first machine learning model identifies the input point on the geometric mesh based on the specified resolution. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the first machine learning model is configured to map any point in any particular resolution to a corresponding actuation value. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, wherein the first machine learning model is resolution-independent. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising training the first machine learning model based on one or more losses associated with the simulated soft body. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the one or more losses comprise a similarity loss that is computed based on the target shape and the simulated soft body. ### Claim 9 (depends on 7) The computer-implemented method of claim 7, wherein training the first machine learning model causes the first machine learning model to learn an implicit function that maps the latent code and the input point on the geometric mesh to one of the simulator control values that specifies the deformation of the geometric mesh at the input point. ### Claim 10 (depends on 7) The computer-implemented method of claim 7, wherein training the first machine learning model comprises: determining a first gradient accumulated on one or more geometric vertices of the simulated soft body; determining, based on the first gradient, a second gradient accumulated on the one or more simulator control values; and updating one or more weights of the first machine learning model based on the second gradient. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein determining, based on the first gradient, the second gradient accumulated on the one or more simulator control values comprises multiplying the first gradient by a sensitivity matrix. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the sensitivity matrix based on a closed-form Hessian of an energy density function. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, wherein the input point is one of a plurality of input points, and the generating, generating, via execution of the first machine learning model, one or more simulator control values is repeated for each of the plurality of input points. ### Claim 14 (depends on 13) The computer-implemented method of claim 13, wherein the first machine learning model comprises an actuation neural network, and the one or more simulator control values comprise an actuation matrix for each of the plurality of input points. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, wherein the first machine learning model comprises a bone position neural network, and the method further comprises: inputting the latent code and the input point on the geometric mesh into the bone position neural network, wherein the input point on the geometric mesh corresponds to a bone region of the target shape; and generating, via execution of the bone position neural network, a transformed bone region position to which the input point on the geometric mesh is to be moved, wherein the one or more simulator control values comprise the transformed bone region position. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein the latent code is a value in a latent control space of the differentiable simulator. ### Claim 17 One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: receiving a latent code that represents a target shape; inputting the latent code and an input point on a geometric mesh associated with the target shape into a first machine learning model; generating, via execution of the first machine learning model, one or more simulator control values that specify a deformation of the geometric mesh, wherein each of the simulator control values is based on the latent code and corresponds to the input point on the geometric mesh; generating, via execution of a differentiable simulator, a simulated soft body based on the one or more simulator control values and the geometric mesh; and causing the simulated soft body to be outputted to a computing device, wherein the simulated soft body is used to perform one or more simulation operations. ### Claim 18 (depends on 17) The one or more non-transitory computer-readable media of claim 17, wherein generating, via execution of the differentiable simulator, the simulated soft body comprises: generating, via execution of the differentiable simulator, an actuated geometric mesh by deforming the geometric mesh based on the one or more simulator control values; wherein the simulated soft body is generated based on the actuated geometric mesh. ### Claim 19 (depends on 17) The one or more non-transitory computer-readable media of claim 17, further comprising: inputting the target shape and a specified resolution into an encoder machine learning model; and generating, via execution of the encoder machine learning model for the specified resolution, the latent code that represents the target shape. ### Claim 20 A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: receiving a latent code that represents a target shape; inputting the latent code and an input point on a geometric mesh associated with the target shape into a first machine learning model; generating, via execution of the first machine learning model, one or more simulator control values that specify a deformation of the geometric mesh, wherein each of the simulator control values is based on the latent code and corresponds to the input point on the geometric mesh; generating, via execution of a differentiable simulator, a simulated soft body based on the one or more simulator control values and the geometric mesh; and causing the simulated soft body to be outputted to a computing device, wherein the simulated soft body is used to perform one or more simulation operations.