# US 12,182,940 B2 · Self-supervised single-view 3D reconstruction via semantic consistency > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,940 B2 - Document ID: US12182940B2 - Status: Granted - Kind: B2 - Application number: 17/578,051 - Filed: 2022-01-18 - Published: 2022-05-05 - Granted: 2024-12-31 - Earliest claimed: 2020-03-13 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Xueting Li; Sifei Liu; Kihwan Kim; Shalini De Mello; Varun Jampani; Jan Kautz - Web page: https://patentgrep.com/patent/US12182940B2 ## CPC - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06T 7/50: Depth or shape recovery - G06T 7/74: {involving reference images or patches} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/776: Validation; Performance evaluation - G06V 10/82: using neural networks - G06V 20/64: Three-dimensional [3D] objects - G06T 2200/08: involving all processing steps from image acquisition to 3D model generation - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Apparatuses, systems, and techniques to identify a shape or camera pose of a three-dimensional object from a two-dimensional image of the object. In at least one embodiment, objects are identified in an image using one or more neural networks that have been trained on objects of a similar category and a three-dimensional mesh template. ## Claims ### Claim 1 A method, comprising: using a neural network to generate a three-dimensional model of an object from a two-dimensional image; segmenting one or more parts of the object on the three-dimensional model to generate a canonical semantic map based, at least in part, on self-supervised co-part segmentation; and identifying the one or more parts of the object on the three-dimensional model using the canonical semantic map. ### Claim 2 (depends on 1) The method of claim 1, wherein the canonical semantic map is generated from a set of two-dimensional images using the self-supervised co-part segmentation. ### Claim 3 (depends on 1) The method of claim 1, wherein the canonical semantic map is a map in UV space that is able to be mapped to the three-dimensional model. ### Claim 4 (depends on 1) The method of claim 1, wherein the canonical semantic map is a category-level semantic map for a type of object. ### Claim 5 (depends on 1) The method of claim 1, further comprising training the neural network by at least comparing consistency between semantic parts in three-dimensional and two-dimensional images. ### Claim 6 (depends on 1) The method of claim 1, further comprising displaying a name associated with the one or more parts. ### Claim 7 (depends on 1) The method of claim 1, further comprising: rendering a reconstructed image from the three-dimensional model of the object, an estimated texture of the object, and a camera pose; comparing the reconstructed image to the two-dimensional image to form a comparison; and modifying the neural network based on differences between the reconstructed image and the two-dimensional image. ### Claim 8 (depends on 1) The method of claim 1, wherein the three-dimensional model is generated based at least in part on an image mask that delineates a silhouette of the object. ### Claim 9 (depends on 1) The method of claim 1, wherein the neural network includes a generative adversarial network. ### Claim 10 A system, comprising memory to store instructions that, as a result of execution by one or more processors, causes the system to: use a neural network to generate a three-dimensional model of an object from a two-dimensional image; segment one or more parts of the object on the three-dimensional model to generate a canonical semantic map based, at least in part, on self-supervised co-part segmentation; and identify the one or more parts of the object on the three-dimensional model using the canonical semantic map. ### Claim 11 (depends on 10) The system of claim 10, wherein the canonical semantic map is generated from a set of two-dimensional images using the self-supervised co-part segmentation. ### Claim 12 (depends on 10) The system of claim 10, wherein the canonical semantic map is a map in UV space that is able to be mapped to the three-dimensional model. ### Claim 13 (depends on 10) The system of claim 10, wherein the canonical semantic map is a category-level semantic map for a type of object. ### Claim 14 (depends on 10) The system of claim 10, wherein the instructions, as a result of execution by the one or more processors, further cause the system to train the neural network by at least comparing consistency between semantic parts in three-dimensional and two-dimensional images. ### Claim 15 (depends on 10) The system of claim 10, wherein the instructions, as a result of execution by the one or more processors, further cause the system to display a name associated with the one or more parts. ### Claim 16 (depends on 10) The system of claim 10, wherein the instructions, as a result of execution by the one or more processors, further cause the system to: render a reconstructed image from the three-dimensional model of the object, an estimated texture of the object, and a camera pose; compare the reconstructed image to the two-dimensional image to form a comparison; and modify the neural network based on differences between the reconstructed image and the two-dimensional image. ### Claim 17 (depends on 10) The system of claim 10, wherein the instructions, as a result of execution by the one or more processors, cause the system to generate the three-dimensional model based at least in part on an image mask that delineates a silhouette of the object. ### Claim 18 (depends on 10) The system of claim 10, wherein the neural network includes a generative adversarial network. ### Claim 19 (depends on 10) The system of claim 10, wherein: the instructions, as a result of execution by the one or more processors, further cause the system to generate an output image from the three-dimensional model using an estimated texture as viewed from an estimated camera pose; and generation of the output image employs a differentiable Tenderer. ### Claim 20 A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to: use a neural network to generate a three-dimensional model of an object from a two-dimensional image; segment one or more parts of the object on the three-dimensional model to generate a canonical semantic map based, at least in part, on self-supervised co-part segmentation; and identify the one or more parts of the object on the three-dimensional model using the canonical semantic map. ### Claim 21 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the canonical semantic map is generated from a set of two-dimensional images using self-supervised learning. ### Claim 22 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the canonical semantic map is a map in UV space that is able to be mapped to the three-dimensional model. ### Claim 23 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the canonical semantic map is a category-level semantic map for a type of object. ### Claim 24 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the instructions, as a result of being performed by the one or more processors, further cause the one or more processors to train the neural network by at least comparing consistency between semantic parts in three-dimensional and two-dimensional images. ### Claim 25 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the instructions further cause the one or more processors to display a name associated with the one or more parts. ### Claim 26 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the instructions, as a result of being performed by the one or more processors, further cause the one or more processors to: render a reconstructed image from the three-dimensional model of the object, an estimated texture of the object, and a camera pose; compare the reconstructed image to the two-dimensional image to form a comparison; and modify the neural network based on differences between the reconstructed image and the two-dimensional image. ### Claim 27 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the three-dimensional model is generated based at least in part on an image mask that delineates a silhouette of the object. ### Claim 28 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein the neural network includes a generative adversarial network. ### Claim 29 (depends on 20) The non-transitory machine-readable medium of claim 20, wherein: the instructions, as a result of being performed by the one or more processors, further cause the one or more processors to generate an output image from the three-dimensional model using an estimated texture as viewed from an estimated camera pose; and generation of the output image employs a differentiable Tenderer.