# US 2022/0044075 A1 · Neural network model trained using generated synthetic images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044075 A1
- Document ID: US20220044075A1
- Status: Published application
- Kind: A1
- Application number: 17/507,620
- Filed: 2021-10-21
- Published: 2022-02-10
- Earliest claimed: 2018-02-14
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Jonathan Tremblay; Aayush Prakash; Mark A. Brophy; Varun Jampani; Cem Anil; Stanley Thomas Birchfield; Thang Hong To; David Jesus Acuna Marrero
- Web page: https://patentgrep.com/patent/US20220044075A1

## CPC

- G06T 15/00: Three-dimensional [3D] image rendering
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 15/04: Texture mapping
- G06T 15/20: Perspective computation
- G06T 15/50: Lighting effects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2210/12: Bounding box
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Training deep neural networks requires a large amount of labeled training data. Conventionally, labeled training data is generated by gathering real images that are manually labelled which is very time-consuming. Instead of manually labelling a training dataset, domain randomization technique is used generate training data that is automatically labeled. The generated training data may be used to train neural networks for object detection and segmentation (labelling) tasks. In an embodiment, the generated training data includes synthetic input images generated by rendering three-dimensional (3D) objects of interest in a 3D scene. In an embodiment, the generated training data includes synthetic input images generated by rendering 3D objects of interest on a 2D background image. The 3D objects of interest are objects that a neural network is trained to detect and/or label.

## Claims

### Claim 1

A processor, comprising:
one or more processing units to receive, using a neural network model, an input image comprising at least one object of interest and an additional object, wherein the neural network is trained to ignore the additional object and to produce task-specific output data corresponding to the object of interest.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the neural network model is trained to ignore the additional object and to produce the task specific output data by using a training dataset that includes synthetic input images constructed from rendered three-dimensional (3D) objects of interest, one or more rendered 3D geometric shapes, and task-specific training data corresponding to the 3D objects of interest.

### Claim 3 (depends on 2)

The processor of claim 2, wherein the one or more rendered 3D geometric shapes are omitted from the task-specific training data.

### Claim 4 (depends on 2)

The processor of claim 2, wherein the 3D objects of interest are rendered according to rendering parameters that comprise at least one of:
data corresponding to a position of a light;
data corresponding to an orientation of a light;
data corresponding to a color of the light; or
data corresponding to an intensity of the light.

### Claim 5 (depends on 2)

The processor of claim 2, wherein the 3D objects of interest are rendered by applying a texture map to at least one of the 3D objects of interest.

### Claim 6 (depends on 2)

The processor of claim 2, wherein the one or more 3D geometric shapes are rendered by applying a texture map to at least one of the one or more 3D geometric shapes.

### Claim 7 (depends on 2)

The processor of claim 2, wherein the task-specific training data is computed and the one or more rendered 3D geometric shapes are omitted from the task-specific training data.

### Claim 8 (depends on 2)

The processor of claim 2, wherein the synthetic input images include a background image of a 3D scene.

### Claim 9 (depends on 1)

The processor of claim 1, wherein the task-specific output data is associated with at least one of an instance segmentation task, an object detection task, or a semantic segmentation task.

### Claim 10 (depends on 1)

The processor of claim 1, wherein the task-specific output data defines a location and dimensions of a bounding shape enclosing the object of interest.

### Claim 11 (depends on 1)

The processor of claim 1, wherein the task-specific output data comprises an object identifier for each pixel that is covered by the object of interest.

### Claim 12 (depends on 1)

The processor of claim 1, wherein the one or more processing units are included in a server or in a data center and the task-specific output data is streamed to a device.

### Claim 13 (depends on 1)

The processor of claim 1, wherein the one or more processing units are included in a server or in a data center and the input image is received from a device.

### Claim 14 (depends on 1)

The processor of claim 1, wherein the neural network model is deployed in a machine, robot, or autonomous vehicle.

### Claim 15

A non-transitory, computer-readable storage medium storing instructions that, when executed by a processing unit, cause the processing unit to receive, using a neural network model, an input image comprising at least one object of interest and an additional object, wherein the neural network model is trained to ignore the additional object and to produce task-specific output data corresponding to the object of interest.

### Claim 16 (depends on 15)

The non-transitory, computer-readable storage medium of claim 15, wherein the neural network model is trained to ignore the additional object and produce the task-specific output data by using a training dataset that includes synthetic input images constructed from rendered three-dimensional (3D) objects of interest, one or more rendered 3D geometric shapes, and task-specific training data corresponding to the 3D objects of interest.

### Claim 17 (depends on 15)

The non-transitory, computer-readable storage medium of claim 15, wherein the task-specific output data is associated with at least one of an instance segmentation task, an object detection task, or a semantic segmentation task.

### Claim 18

A computer-implemented method, comprising:
receiving, by a neural network model, an input image comprising at least one object of interest and an additional object; and
processing the input image, by the neural network model, wherein the neural network model is trained to ignore the additional object and produce task-specific output data corresponding to the object of interest.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the neural network model is trained using a training dataset that includes synthetic input images constructed from rendered three-dimensional (3D) objects of interest, one or more rendered 3D geometric shapes, and task-specific training data corresponding to the 3D objects of interest.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, wherein the one or more rendered 3D geometric shapes are omitted from the task-specific training data.
