# US 12,183,037 B2 · 3D pose estimation in robotics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,037 B2 - Document ID: US12183037B2 - Status: Granted - Kind: B2 - Application number: 18/487,800 - Filed: 2023-10-16 - Published: 2024-02-01 - Granted: 2024-12-31 - Earliest claimed: 2021-03-03 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Sravya Nimmagadda; David Weikersdorfer - Web page: https://patentgrep.com/patent/US12183037B2 ## CPC - G06T 7/74: {involving reference images or patches} - G06T 7/75: {involving models} - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1697: {Vision controlled systems} - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 3/096: Transfer learning - G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G05B 2219/40527: Modeling, identification of link parameters - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An autoencoder may be trained to predict 3D pose labels using simulation data extracted from a simulated environment, which may be configured to represent an environment in which the 3D pose estimator is to be deployed. Assets may be used to mimic the deployment environment such as 3D models or textures and parameters used to define deployment scenarios and/or conditions that the 3D pose estimator will operate under in the environment. The autoencoder may be trained to predict a segmentation image from an input image that is invariant to occlusions. Further, the autoencoder may be trained to exclude areas of the input image from the object that correspond to one or more appendages of the object. The 3D pose may be adapted to unlabeled real-world data using a GAN, which predicts whether output of the 3D pose estimator was generated from real-world data or simulated data. ## Claims ### Claim 1 At least one processor comprising: one or more circuits to: receive image data capturing an object in a field of view of at least one sensor associated with a robot or autonomous machine; generate, using one or more Machine Learning Models (MLMs) and the image data, predictions indicating one or more pixels of an image depicting one or more portions of the object to exclude from a segmentation mask that corresponds to the object based at least on the one or more portions corresponding to one or more appendages of the object; and determine one or more parameters corresponding to a pose of the object based at least on the predictions; and perform one or more control operations for the robot or autonomous machine based at least on the one or more parameters. ### Claim 2 (depends on 1) The at least one processor of claim 1, wherein the predictions indicate the one or more pixels to exclude from the segmentation mask based at least on the one or more MLMs detecting the one or more pixels correspond to the one or more appendages. ### Claim 3 (depends on 1) The at least one processor of claim 1, wherein the determining the one or more parameters corresponding to the pose of the object is based at least on updating, using the predictions, an initial pose determined for the object. ### Claim 4 (depends on 1) The at least one processor of claim 1, wherein the one or more parameters indicate one or more of an orientation of the object or a position of the object. ### Claim 5 (depends on 1) The at least one processor of claim 1, wherein the one or more circuits are further to determine, using an object detector, one or more locations of the object in an image, wherein the image data corresponds to a version of the image that is modified based at least on the one or more locations. ### Claim 6 (depends on 1) The at least one processor of claim 1, wherein the one or more MLMs are updated to generate the predictions using ground truth data corresponding to segmentation masks produced from data representing a simulated environment comprising a model of the object, the segmentation masks excluding the one or more portions of the one or more appendages of the object. ### Claim 7 (depends on 1) The at least one processor of claim 1, wherein the one or more MLMS decode a representation of an image corresponding to the image data into the predictions corresponding to the segmentation mask. ### Claim 8 (depends on 1) The at least one processor of claim 1, wherein the at least one processor is comprised in at least one of: a control system for the robot or autonomous machine; a perception system for the robot or autonomous machine; a system for performing simulation operations; a system for performing light transport simulation; a system for performing content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. ### Claim 9 A method comprising: receiving an image capturing an object in a field of view of at least one sensor; receiving ground truth data representing one or more parameters corresponding to a pose of the object in the image and indicating one or more pixels that correspond to one or more portions of one or more appendages of the object in a segmentation mask corresponding to the image correspond to a background of the object based at least on the one or more portions corresponding to the one or more appendages of the object; and updating, using the image and the ground truth data, one or more Machine Learning Models (MLMs) to generate predictions indicating the one or more pixels that correspond to the one or more portions of one or more appendages of the object in the segmentation mask correspond to the background of the object and predictions indicating the one or more parameters corresponding to the pose. ### Claim 10 (depends on 9) The method of claim 9, wherein the image corresponds to a render of a field of perception corresponding to the at least one sensor in a virtual environment. ### Claim 11 (depends on 9) The method of claim 9, wherein the segmentation mask is rendered from three-dimensional (3D) data that represents a simulated environment comprising the object. ### Claim 12 (depends on 9) The method of claim 9, wherein at least one appendage of the one or more appendages has a different orientation than the pose of the object. ### Claim 13 (depends on 9) The method of claim 9, wherein the one or more parameters indicate one or more of an orientation of the object or a position of the object. ### Claim 14 (depends on 9) The method of claim 9, wherein the one or more MLMS are updated to decode the image into the predictions corresponding to the segmentation mask. ### Claim 15 A system comprising: one or more processors to perform a method comprising: receiving, image data capturing an object in a field of view of at least one sensor; generating, using the image data and one or more Machine Learning Models (MLMs), predictions that ignore or minimize one or more pixels depicting at least a portion of one or more appendages of the object based at least on the one or more pixels corresponding to the one or more appendages of the object, computing, based at least on the predictions, one or more parameters corresponding to a pose of the object; and performing one or more control operations for a robot or autonomous machine based at least on the pose. ### Claim 16 (depends on 15) The system of claim 15, wherein the predictions correspond to a version of an image that excludes the one or more pixels from the image. ### Claim 17 (depends on 16) The system of claim 16, wherein the version of the image includes a segmentation mask. ### Claim 18 (depends on 15) The system of claim 15, wherein the predictions ignore or minimize the one or more pixels based at least on the one or more MLMs detecting the one or more pixels correspond to the one or more appendages. ### Claim 19 (depends on 15) The system of claim 15, wherein the one or more MLMs are trained to predict the one or more pixels representing the one or more appendages corresponds to a background of the object. ### Claim 20 (depends on 15) The system of claim 15, wherein the one or more processors are comprised in at least one of: a control system for the robot or autonomous machine; a perception system for the robot or autonomous machine; a system for performing simulation operations; a system for performing light transport simulation; a system for performing content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.