# US 2023/0274492 A1 · Texture transfer and synthesis using aligned maps in image generation systems and applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274492 A1
- Document ID: US20230274492A1
- Status: Published application
- Kind: A1
- Application number: 18/149,454
- Filed: 2023-01-03
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Zhiqin Chen; Kangxue Yin; Sanja Fidler
- Web page: https://patentgrep.com/patent/US20230274492A1

## CPC

- G06T 15/04: Texture mapping
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Approaches presented herein can utilize a network that learns to embed three-dimensional (3D) coordinates on a surface of one or more 3D shapes into an aligned two-dimensional (2D) texture space, where corresponding parts of different 3D shapes can be mapped to the same location in a texture image. Alignment can be performed using a texture alignment module that generates a set of basis images for synthesizing textures. A trained network can generate a basis shared by all shape textures, and can predict input-specific coefficients to construct the output texture for each shape as a linear combination of the basis images, then deform the texture to match the pose of the input. Such an approach can ensure alignment of textures, even in situations with at least somewhat limited network capacity. To unwrap shapes of complex structure or topology, a masking network can be utilized that cuts the shape into multiple pieces to reduce the distortion in the 2D mapping.

## Claims

### Claim 1

A computer-implemented method, comprising:
obtaining a three-dimensional (3D) representation of a first object;
generating one or more feature encodings corresponding to the 3D representation;
mapping coordinates of the 3D representation to a two-dimensional (2D) texture space;
generating, based in part on the mapped coordinates in the 2D texture space and the one or more feature encodings, a 2D texture image corresponding to the input 3D representation of the first object; and
deforming the 2D texture image to correspond to 3D geometry of a second object.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D representation of the first object is a point cloud comprising points with color data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more feature encodings include a first texture feature vector corresponding to a first portion of the first object, a second texture feature vector corresponding to a second portion of the first object, and a shape code vector generated by an encoder network receiving at least the 3D representation of the first object as input.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
providing the mapped coordinates in the 2D texture space as input to a first basis generator network, to generate a first texture basis, and a second basis generator network to generate a second texture basis.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising:
generating a first texture image using point color data determined based in part on the first texture basis and the first texture feature vector; and
generating a second texture image using point cloud color data determined based in part on the second texture basis and the second texture feature vector.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
segmenting surface points of the 3D representation of the first object to generate a first mask, corresponding to the first portion of the first object, and a second mask corresponding to the second portion of the first object; and
using the first mask and the second mask with the first texture image and the second texture image to generate a masked combination of color values for coordinates of the second object.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the 3D representation of the first object is segmented into a plurality of voxels before the one or more feature encodings are generated by an encoding network.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the target object is a different object from the first object, or the first object in a different pose or state.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more feature encodings corresponding to the first object include semantic information for the object.

### Claim 10

A processor, comprising:
one or more circuits to:
obtain a three-dimensional (3D) representation of a first object;
generate one or more feature encodings corresponding to the 3D representation;
map coordinates of the 3D representation to a two-dimensional (2D) texture space;
generate, based in part on the mapped coordinates in the 2D texture space and the one or more feature encodings, a 2D texture image corresponding to the input 3D representation of the first object; and
deform the 2D texture image to correspond to 3D geometry of a second object.

### Claim 11 (depends on 10)

The processor of claim 10, wherein the one or more feature encodings include a first texture feature vector corresponding to a first portion of the first object, a second texture feature vector corresponding to a second portion of the first object, and a shape code vector generated by an encoder network receiving at least the 3D representation of the first object as input.

### Claim 12 (depends on 11)

The processor of claim 11, wherein the one or more circuits are to perform operations further to:
provide the mapped coordinates in the 2D texture space as input to a first basis generator network, to generate a first texture basis, and a second basis generator network to generate a second texture basis.

### Claim 13 (depends on 12)

The processor of claim 12, wherein the one or more circuits are to perform operations further to:
generate a first texture image using point color data determined based in part on the first texture basis and the first texture feature vector; and
generate a second texture image using point cloud color data determined based in part on the second texture basis and the second texture feature vector.

### Claim 14 (depends on 13)

The processor of claim 13, wherein the one or more circuits are to perform operations further to:
segment surface points of the 3D representation of the first object to generate a first mask, corresponding to the first portion of the first object, and a second mask corresponding to the second portion of the first object; and
use the first mask and the second mask with the first texture image and the second texture image to generate a masked combination of color values for coordinates of the second object.

### Claim 15 (depends on 10)

The processor of claim 10, wherein the processor is comprised in at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for synthetic data generation;
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.

### Claim 16

A system, comprising:
one or more processors to use a neural network to deform at least one 2D texture image of a first object to correspond to a 3D geometry of a second object, the one or more processors further to generate the at least one 2D texture image based in part on mapped coordinates of a 3D representation of the first object in 2D texture space, and one or more feature encodings extracted from the 3D representation.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more feature encodings include a first texture feature vector corresponding to a first portion of the first object, a second texture feature vector corresponding to a second portion of the first object, and a shape code vector generated by an encoder network receiving at least the 3D representation of the first object as input.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more processors are further to:
provide the mapped coordinates in the 2D texture space as input to a first basis generator network, to generate a first texture basis, and a second basis generator network to generate a second texture basis.

### Claim 19 (depends on 18)

The system of claim 18, wherein the one or more processors are further to:
generate a first texture image using point color data determined based in part on the first texture basis and the first texture feature vector; and
generate a second texture image using point cloud color data determined based in part on the second texture basis and the second texture feature vector.

### Claim 20 (depends on 16)

The system of claim 16, wherein the system comprises at least one of:
a system for performing simulation operations;
a system for performing simulation operations to test or validate autonomous machine applications;
a system for performing digital twin operations;
a system for performing light transport simulation;
a system for rendering graphical output;
a system for performing deep learning operations;
a system implemented using an edge device;
a system for generating or presenting virtual reality (VR) content;
a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content;
a system incorporating one or more Virtual Machines (VMs);
a system implemented at least partially in a data center;
a system for performing hardware testing using simulation;
a system for synthetic data generation;
a collaborative content creation platform for 3D assets; or
a system implemented at least partially using cloud computing resources.
