# US 2021/0279952 A1 · Neural rendering for inverse graphics generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279952 A1
- Document ID: US20210279952A1
- Status: Published application
- Kind: A1
- Application number: 17/193,405
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Wenzheng Chen; Yuxuan Zhang; Sanja Fidler; Huan Ling; Jun Gao; Antonio Torralba Barriuso
- Web page: https://patentgrep.com/patent/US20210279952A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 15/10: Geometric effects
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Approaches are presented for training an inverse graphics network. An image synthesis network can generate training data for an inverse graphics network. In turn, the inverse graphics network can teach the synthesis network about the physical three-dimensional (3D) controls. Such an approach can provide for accurate 3D reconstruction of objects from 2D images using the trained inverse graphics network, while requiring little annotation of the provided training data. Such an approach can extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers, enabling a disentangled generative model to function as a controllable 3D “neural renderer,” complementing traditional graphics renderers.

## Claims

### Claim 1

A computer-implemented method, comprising:
providing a two-dimensional image of an object as input to a generative network;
generating, using the generative network, a set of view images of the object represented from different views;
providing, as input to an inverse graphics network, the set of view images and information for the different views;
determining, for individual view images of the set using the inverse graphics network, a set of three-dimensional information;
rendering, for the individual view images of the set, a representation of the object using the set of three-dimensional information and the respective view information;
comparing the rendered representations against corresponding data to determine at least one loss value; and
adjusting one or more network parameters for the inverse graphics network based at least in part upon the at least one loss value.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing at least a subset of the representations of the object, rendered by the inverse graphics network, as training data to further train the generative network.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
training the inverse graphics network and the generative network together using a common loss function.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the generative network is a style generative adversarial network enabling only camera view-related features to be adjusted for generating the set of view images.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
using a selection matrix to reduce a dimensionality of image features to be included in a latent code to be used to render the representation of the object.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
rendering the representation of the object based, at least in part, upon the latent code and using a differentiable renderer.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein the latent code includes camera features for the corresponding view.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the three-dimensional information for the object includes at least one of a shape, texture, lighting, or background for the object.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the two-dimensional image input to the generative network is annotated with weakly accurate camera information corresponding to a subset of object features.

### Claim 10

A system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
provide a two-dimensional image of an object as input to a generative network;
generate, using the generative network, a set of view images of the object represented from different views;
provide, as input to an inverse graphics network, the set of view images and information for the different views;
determine, for individual view images of the set using the inverse graphics network, a set of three-dimensional information;
render, for the individual view images of the set, a representation of the object using the set of three-dimensional information and the respective view information;
compare the rendered representations against corresponding data to determine at least one loss value; and
adjust one or more network parameters for the inverse graphics network based at least in part upon the at least one loss value.

### Claim 11 (depends on 10)

The system of claim 10, wherein the instructions when executed further cause the system to:
provide at least a subset of the representations of the object, rendered by the inverse graphics network, as training data to further train the generative network.

### Claim 12 (depends on 11)

The system of claim 11, wherein the instructions when executed further cause the system to:
train the inverse graphics network and the generative network together using a common loss function.

### Claim 13 (depends on 10)

The system of claim 10, wherein the generative network is a style generative adversarial network enabling only camera view-related features to be adjusted for generating the set of view images.

### Claim 14 (depends on 10)

The system of claim 10, wherein the instructions when executed further cause the system to:
use a selection matrix to reduce a dimensionality of image features to be included in a latent code to be used to render the representation of the object; and
render the representation of the object based, at least in part, upon the latent code using a differentiable renderer.

### Claim 15 (depends on 10)

The system of claim 10, wherein the system comprises at least one of:
a system for performing graphical rendering operations;
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing deep learning operations;
a system implemented using an edge device;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center; or
a system implemented at least partially using cloud computing resources.

### Claim 16

A computer-implemented method, comprising:
receiving a two-dimensional image; and
generating a three-dimensional representation of the two-dimensional image using an inverse graphics network, the inverse graphics network trained at least in part by:
generating, using a generative network and a two-dimensional input image of an object, a set of view images of the object represented from different views;
determining, for individual view images of the set using the inverse graphics network, a set of three-dimensional information;
rendering, for the individual view images of the set, a representation of the object using the set of three-dimensional information and information for the respective view;
comparing the rendered representations against corresponding representations to determine at least one loss value; and
adjusting one or more network parameters for the inverse graphics network based at least in part upon the at least one loss value.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, further comprising:
providing at least a subset of the representations of the object, rendered by the inverse graphics network, as training data to further train the generative network.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, further comprising:
training the inverse graphics network and the generative network together using a common loss function.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16, wherein the generative network is a style generative adversarial network enabling only camera view-related features to be adjusted for generating the set of view images.

### Claim 20 (depends on 16)

The computer-implemented method of claim 16, wherein the two-dimensional input image is annotated with weakly accurate camera information corresponding to a subset of object features.