# US 2022/0284624 A1 · 3d pose estimation in robotics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284624 A1
- Document ID: US20220284624A1
- Status: Published application
- Kind: A1
- Application number: 17/191,543
- Filed: 2021-03-03
- Published: 2022-09-08
- Earliest claimed: 2021-03-03
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Sravya Nimmagadda; David Weikersdorfer
- Web page: https://patentgrep.com/patent/US20220284624A1

## CPC

- G06T 7/74: {involving reference images or patches}
- G06T 7/75: {involving models}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1697: {Vision controlled systems}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G05B 2219/40527: Modeling, identification of link parameters
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

An autoencoder may be trained to predict 3D pose labels using simulation data extracted from a simulated environment, which may be configured to represent an environment in which the 3D pose estimator is to be deployed. Assets may be used to mimic the deployment environment such as 3D models or textures and parameters used to define deployment scenarios and/or conditions that the 3D pose estimator will operate under in the environment. The autoencoder may be trained to predict a segmentation image from an input image that is invariant to occlusions. Further, the autoencoder may be trained to exclude areas of the input image from the object that correspond to one or more appendages of the object. The 3D pose may be adapted to unlabeled real-world data using a GAN, which predicts whether output of the 3D pose estimator was generated from real-world data or simulated data.

## Claims

### Claim 1

A processor comprising:
one or more circuits to:
provide, as input to one or more Machine Learning Models (MLMs), image data representing a first portion of an object in a field of view of at least one sensor in an environment and an occlusion to a second portion of the object;
receive, from the one or more MLMs, first output data representing a Three-Dimensional (3D) pose of the object in the environment and second output data representing predictions that one or more pixels in a segmentation mask correspond to at least the second portion of the object; and
determine the 3D pose using the first output data.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to transmit data that causes a robot to be controlled using the 3D pose.

### Claim 3 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to determine, using an object detector, one or more locations of the object in an image, wherein the image data represents a version of the image that is modified based at least on the one or more locations.

### Claim 4 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to:
generate the segmentation mask using the second output data; and
refine the 3D pose using the segmentation mask.

### Claim 5 (depends on 1)

The processor of claim 1, wherein the image data further represents a second portion of the object that depicts an appendage of the object and the one or more MLMs are trained to exclude the appendage from the object in the segmentation mask.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more MLMs was trained to predict the 3D pose using inputs comprising 3D renders of a model of the object in a simulated environment and using data corresponding to 3D pose labels of the object extracted from data that defines 3D poses of the model in the simulated environment.

### Claim 7 (depends on 1)

The processor of claim 1, wherein the one or more MLMs was trained to generate the predictions that the one or more pixels correspond to the at least the second portion of the object in the segmentation mask using ground truth corresponding to segmentation masks rendered from data representing a simulated environment comprising a model of the object.

### Claim 8 (depends on 1)

The processor of claim 1, wherein the environment is a physical environment and the one or more MLMs was trained to predict the 3D pose using inputs comprising one or more 3D renders of a model of the object in a virtual environment configured to simulate the physical environment.

### Claim 9

A method comprising:
receiving image data representing at least a portion of a Three-Dimensional (3D) render of a field of view of at least one sensor in a virtual environment, the 3D render depicting a first portion of an object in the field of view and an occlusion to a second portion of the object;
receiving first data representing a 3D pose of the object in the 3D render of the virtual environment;
receiving second data identifying one or more pixels in a segmentation mask that correspond to at least the second portion of the object; and
training one or more Machine Learning Models (MLMs) to predict that the one or more pixels in the segmentation mask correspond to the at least the second portion of the object and predict the 3D pose of the object from at least a portion of the 3D render using the first data, the second data, and the image data.

### Claim 10 (depends on 9)

The method of claim 9, wherein the first data is extracted from data that defines the 3D pose of the object in the simulated environment.

### Claim 11 (depends on 9)

The method of claim 9, wherein the second data is extracted from the segmentation mask and the segmentation mask is rendered from 3D data that represents the simulated environment.

### Claim 12 (depends on 9)

The method of claim 9, further comprising:
receiving input that identifies graphical assets in association with one or more elements of a 3D environment and defines at least one parameter of one or more properties of the simulated environment;
generating a configuration of the simulated environment that uses the graphical assets to represent the one or more items in the 3D environment based at least on selecting the 3D pose for the object and selecting a value of the one or more properties using the at least one parameter; and
rendering at least the 3D render using the configuration of the simulated environment.

### Claim 13 (depends on 9)

The method of claim 9, wherein the 3D render depicts an appendage of the object and the one or more MLMs are trained to exclude the appendage from the object in the segmentation mask.

### Claim 14 (depends on 9)

The method of claim 9, where the receiving of the image data is based at least on configuring the virtual environment to simulate a physical environment.

### Claim 15

A system comprising:
one or more processors; and
one or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method comprising:
receiving input that identifies graphical assets in association with one or more elements of a Three-Dimensional (3D) environment and defines at least one parameter of one or more properties of the 3D environment;
generating a configuration of the 3D environment that uses the graphical assets to represent the one or more elements in the 3D environment based at least on selecting a 3D pose for an object and a value of the one or more properties using the at least one parameter;
rendering an image depicting the 3D environment in the configuration from a perspective of one or more virtual sensors in the 3D environment; and
providing the image as training data used to train one or more MLMs to predict 3D poses from image data.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more elements comprise the object and other objects used to populate the 3D environment.

### Claim 17 (depends on 15)

The system of claim 15, wherein the graphical assets comprise one or more 3D models and one or more textures of the 3D models.

### Claim 18 (depends on 15)

The system of claim 15, wherein the one or more elements comprise one or more objects and the one or more properties comprise a quantity of the one or more objects spawned in the 3D environment.

### Claim 19 (depends on 15)

The system of claim 15, wherein the one or more properties comprise lighting conditions of the 3D environment.

### Claim 20 (depends on 15)

The system of claim 15, wherein the one or more properties comprise a height of the one or more virtual sensors in the 3D environment.

### Claim 21 (depends on 15)

The system of claim 15, wherein the method further includes:
selecting from unlabeled real-world data and labeled simulated data as an input to a generator comprising at least an encoder of the one or more MLMs;
providing a latent code generated by the encoder from the input to the generator as an input to a discriminator model, wherein the discriminator model generates a prediction on whether the latent code corresponds to the real-world data or the labeled simulated data; and
updating one or more portions of the one or more MLMs based at least on the prediction.
