# US 2024/0257437 A1 · Real-time neural appearance models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257437 A1
- Document ID: US20240257437A1
- Status: Published application
- Kind: A1
- Application number: 18/418,680
- Filed: 2024-01-22
- Published: 2024-08-01
- Earliest claimed: 2023-01-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Karthik Vaidyanathan; Alex John Bauld Evans; Jan Novák; Andrea Weidlich; Fabrice Pierre Armand Rousselle; Aaron Eliot Lefohn; Franz Petrik Clarberg; Benedikt Bitterli; Tizian Lucien Zeltner
- Web page: https://patentgrep.com/patent/US20240257437A1

## CPC

- G06T 15/06: Ray-tracing
- G06T 7/33: using feature-based methods
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 15/506: {Illumination models}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments of the present disclosure relate to real-time neural appearance models. Using a neural decoder, scenes are rendered in real-time with complex material appearance previously reserved for offline use. Learned hierarchical textures representing the material properties are encoded as latent codes. When a ray is cast and intersects with geometry in the scene, the intersection point is mapped to one of the latent codes. The latent code is interpreted using neural decoders, which produce reflectance values and importance-sampled directions that can be used to determine a pixel color.

## Claims

### Claim 1

A computer-implemented method, comprising:
Receiving a latent code defining properties of a material associated with a surface at a point intersected by a ray;
extracting, from the latent code, at least one of the properties of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, reflectance attributes for the surface based on the latent code and the transformed properties.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the material comprises at least one layer including one or more of a low-roughness dielectric coating, conductor with glints, or stains.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising processing the surface incident vector, normal and tangent vectors at the point intersected by the ray, and the latent code by a neural sampling decoder to predict a next segment of a path that includes the ray.]

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising predicting directional albedo, directionless albedo, roughness estimation, or transmissivity for the surface based on the latent code and the transformed properties.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising obtaining the latent code from a stored texture using coordinates of the point.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising processing, by a material encoder, at least one of coordinates and surface parameters of the point intersected by the ray according to learned parameters to generate the latent code.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising training the material encoder by:
determining sample points on the surface;
at each sample point of the sample points, processing the properties of the material, by a material encoder, according to learned parameters to generate estimated latent codes; and
processing the estimated latent codes to predict estimated reflectance attributes at the sample points.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising adjusting the learned parameters based on differences between the estimated reflectance attributes and reference reflectance attributes.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, further comprising adjusting, based on differences between the estimated reflectance attributes and reference reflectance attributes, decoding parameters that are applied to the estimated latent codes by the neural material decoder to predict the estimated reflectance attributes.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the properties of the material comprise albedo, roughness, a normal vector, and a tangent vector.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the material comprises at least one volumetric layer.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein matrix operations implemented by the attribute decoder neural network are executed by a tensor core.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed on a server or in a data center to generate an image, and the image is streamed to a user device.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed within a cloud computing environment.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of receiving, extracting, transforming, or predicting is performed on a virtual machine comprising a portion of a graphics processing unit.

### Claim 17

A system, comprising:
a memory that stores a surface; and
a processor that is connected to the memory, wherein the processor is configured to predict reflectance attributes for the surface by:
receiving a latent code defining properties of a material associated with the surface at a point intersected by a ray;
extracting, from the latent code, at least one of the properties of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, the reflectance attributes for the surface based on the latent code and the transformed properties.

### Claim 18 (depends on 17)

The system of claim 17, further comprising obtaining the latent code from a stored texture using coordinates of the point.

### Claim 19

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving a latent code defining properties of a material associated with a surface at a point intersected by a ray;
extracting, from the latent code, at least one of the properties of the material;
transforming a surface incident vector and a surface outgoing vector at the point using the at least one extracted property to produce transformed properties; and
predicting, by an attribute decoder neural network, reflectance attributes for the surface based on the latent code and the transformed properties.

### Claim 20 (depends on 19)

The non-transitory computer-readable media of claim 19, further comprising processing the surface incident vector, normal and tangent vectors at the point intersected by the ray, and the latent code by a neural sampling decoder to predict a next segment of a path that includes the ray.
