# US 2024/0005604 A1 · Synthesizing three-dimensional shapes using latent diffusion models in content generation systems and applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005604 A1
- Document ID: US20240005604A1
- Status: Published application
- Kind: A1
- Application number: 18/320,716
- Filed: 2023-05-19
- Published: 2024-01-04
- Earliest claimed: 2022-05-19
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Karsten Julian Kreis; Xiaohui Zeng; Arash Vahdat; Francis Williams; Zan Gojcic; Or Litany; Sanja Fidler
- Web page: https://patentgrep.com/patent/US20240005604A1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Approaches presented herein provide for the unconditional generation of novel three dimensional (3D) object shape representations, such as point clouds or meshes. In at least one embodiment, a first denoising diffusion model (DDM) can be trained to synthesize a 1D shape latent from Gaussian noise, and a second DDM can be trained to generate a set of latent points conditioned on this 1D shape latent. The shape latent and set of latent points can be provided to a decoder to generate a 3D point cloud representative of a random object from among the object classes on which the models were trained. A surface reconstruction process may be used to generate a surface mesh from this generated point cloud. Such an approach can scale to complex and/or multimodal distributions, and can be highly flexible as it can be adapted to various tasks such as multimodal voxel- or text-guided synthesis.

## Claims

### Claim 1

A computer-implemented method, comprising:
generating, using a first generative diffusion model, a shape latent representing a shape of a three-dimensional object;
generating, using a second generative diffusion model and the shape latent, a set of latent points representative of latent features of the three-dimensional object;
providing the shape latent and the set of latent points as input to a decoder network; and
receiving, from the decoder network, a point cloud comprising a set of points representative of the three-dimensional object.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating, using the point cloud, a three-dimensional mesh for use in rendering a two-dimensional image of the three-dimensional object.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing the shape latent as a conditioning input to the second generative diffusion model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing Gaussian noise as input to the first generative diffusion model.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the shape latent is a one-dimensional, vector-valued global shape latent.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training the first diffusion network using a set of shape latents of a first latent space generated using a hierarchical variational autoencoder (VAE) trained to generate shape latents from a set of input point clouds.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the hierarchical variational autoencoder (VAE) is further trained to generate latent point clouds from the set of input point clouds, the method further comprising:
training the second diffusion network using a set of latent point clouds of a second latent space generated using the hierarchical variational autoencoder (VAE).

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the three-dimensional object is determined unconditionally from one of a set of object classes on which at least the first diffusion network was trained.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing, as input to an encoder, a voxel-based representation of the three-dimensional object in order to condition at least the first diffusion network to generate the shape latent approximating the voxel-based representation.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing, as input to an encoder, a noisy input shape in order to condition at least the first diffusion network to generate the shape latent approximating the noisy input shape.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
providing, as input to the first diffusion network, a text encoding in order to condition at least the first diffusion network to generate the shape latent based in part on text used to generate the text encoding.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, further comprising:
manipulating one or more of the shape latent or the latent point cloud in order to modify the point cloud to be received from the decoder.

### Claim 13

A processor, comprising:
one or more circuits to:
generate, using a first generative diffusion model, a shape latent representing a shape of a three-dimensional object;
generate, using a second generative diffusion model and the shape latent, a latent point cloud representative of latent features of the three-dimensional object;
provide the shape latent and the latent point cloud as input to a decoder network; and
receive, from the decoder network, a point cloud comprising a set of points representative of the three-dimensional object.

### Claim 14 (depends on 13)

The processor of claim 13, wherein the one or more circuits are further to provide the shape latent as a conditioning input to the second generative diffusion model, wherein the shape latent is a one-dimensional, vector-valued global shape latent.

### Claim 15 (depends on 13)

The processor of claim 13, wherein the one or more circuits are further to train the first diffusion network using a set of shape latents of a first latent space, and to train the second diffusion network using a set of latent point clouds of a second latent space, generated using a hierarchical variational autoencoder (VAE).

### Claim 16 (depends on 13)

The processor of claim 13, wherein the processor is comprised in at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for performing generative content operations using a language model;
a system for synthetic data generation;
a system for performing generative AI operations using a large language model (LLM),
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.

### Claim 17

A system, comprising:
one or more processors to generate a point cloud representing a random three-dimensional object from a set of object classes, the point cloud generated using a shape latent determined using a first generative diffusion network and a set of latent points determined using a second generative diffusion network.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more processors are further to provide the shape latent as a conditioning input to the second generative diffusion model, wherein the shape latent is a one-dimensional, vector-valued global shape latent.

### Claim 19 (depends on 17)

The system of claim 17, wherein the one or more processors are further to train the first diffusion network using a set of shape latents of a first latent space, and to train the second diffusion network using a set of latent point clouds of a second latent space, generated using a hierarchical variational autoencoder (VAE).

### Claim 20 (depends on 17)

The system of claim 17, wherein the system comprises at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system for performing generative AI operations using a large language model (LLM),
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for performing generative content operations using a language model;
a system for synthetic data generation;
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.
