# US 2022/0284658 A1 · Fully-fused neural network execution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284658 A1
- Document ID: US20220284658A1
- Status: Published application
- Kind: A1
- Application number: 17/340,283
- Filed: 2021-06-07
- Published: 2022-09-08
- Earliest claimed: 2021-03-03
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Thomas Müller; Nikolaus Binder; Fabrice Pierre Armand Rousselle; Jan Novák; Alexander Georg Keller
- Web page: https://patentgrep.com/patent/US20220284658A1

## CPC

- G06T 15/06: Ray-tracing
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/10: Interfaces, programming languages or software development kits, e.g. for simulating neural networks
- G06T 15/005: {General purpose rendering architectures}
- G06T 15/506: {Illumination models}
- G06N 3/048: Activation functions
- G06T 2210/52: Parallel processing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A fully-connected neural network may be configured for execution by a processor as a fully-fused neural network by limiting slow global memory accesses to reading and writing inputs to and outputs from the fully-connected neural network. The computational cost of fully-connected neural networks scale quadratically with its width, whereas its memory traffic scales linearly. Modern graphics processing units typically have much greater computational throughput compared with memory bandwidth, so that for narrow, fully-connected neural networks, the linear memory traffic is the bottleneck. The key to improving performance of the fully-connected neural network is to minimize traffic to slow “global” memory (off-chip memory and high-level caches) and to fully utilize fast on-chip memory (low-level caches, “shared” memory, and registers), which is achieved by the fully-fused approach. A real-time neural radiance caching technique for path-traced global illumination is implemented using the fully-fused neural network for caching scattered radiance components of global illumination.

## Claims

### Claim 1

A computer-implemented method, comprising:
partitioning inputs to a neural network model into portions that are each sized for execution in a processing core by a single thread block;
loading weights for the neural network model into a register file within a processor once to process the inputs;
independently processing the portions in parallel by a set of processing cores within the processor, wherein weights for a first layer of the neural network model are applied to each of the portions to generate intermediate results for each portion;
storing the intermediate results for each portion into a memory that is shared between the set of processing cores; and
processing the intermediate results for each portion by a subsequent layer of the neural network model to produce subsequent intermediate results until a last layer of the neural network model generates outputs.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the set of processing cores and the memory are included within a graphics processing unit.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the memory comprises at least one of low-level caches, shared on-chip memory, and registers.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the neural network model implements a radiance cache for performing light transport path tracing.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the inputs are three-dimensional (3D) positions associated with light transport paths through a scene and the outputs are radiance predictions at the 3D positions.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
generating an image of the scene corresponding to at least one camera using the radiance predictions;
extending a subset of the light transport paths to generate additional radiance predictions; and
updating the weights based on the additional radiance predictions and the additional radiance prediction propagated to the 3D positions for the subset of the light transport paths.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein each of the light transport paths terminates at the 3D position based on a first size of a first footprint of the light transport path at the 3D position compared with a second size of a second footprint of the light transport path at a visible 3D position.

### Claim 8 (depends on 5)

The computer-implemented method of claim 5, wherein the inputs include at least one of a material property, view direction, or normal vector associated with each of the 3D positions.

### Claim 9 (depends on 5)

The computer-implemented method of claim 5, wherein a filtered copy of the weights is used to generate the radiance predictions at the 3D positions.

### Claim 10 (depends on 6)

The computer-implemented method of claim 6, further comprising:
processing, by the radiance cache, a second 3D position associated with a second light transport path through the scene according to the updated weights to produce a second radiance prediction at the second 3D position; and
generating a second image of the scene using the second radiance prediction.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein at least one of the camera, lighting, geometry, and materials is changed for the second light transport path compared with the light transport paths.

### Claim 12 (depends on 6)

The computer-implemented method of claim 6, wherein at least a portion of the light transport paths are associated with additional cameras.

### Claim 13 (depends on 5)

The computer-implemented method of claim 5, wherein the 3D positions are generated by at least one of a rasterizer, a ray tracer, a ray marcher, a sphere tracer, and a cone tracer.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the neural network model is trained on a server or in a data center and the outputs are streamed to a user device.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed within a cloud computing environment.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed on a server or in a data center and the image is streamed to a user device.

### Claim 17 (depends on 1)

The computer-implemented method of claim 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed on a server or in a data center and the neural network model is streamed to a user device.

### Claim 18 (depends on 1)

The computer-implemented method of claim 1, wherein the outputs are used for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

### Claim 19 (depends on 1)

The computer-implemented method of claim 1, wherein one or more of the steps of partitioning, loading, independently storing, and processing are performed on a virtual machine comprising a portion of a graphics processing unit.

### Claim 20

A system, comprising:
a global memory storing weights;
a processor that is connected to the global memory and executes a neural network model by:
partitioning inputs to the neural network model into portions that are each sized for execution in a processing core by a single thread block;
loading the weights for the neural network model into a register file within the processor once to process the inputs;
independently processing the portions in parallel by a set of processing cores within the processor, wherein weights for a first layer of the neural network model are applied to each of the portions to generate intermediate results for each portion;
storing the intermediate results for each portion into a memory that is shared between the set of processing cores; and
processing the intermediate results for each portion by a subsequent layer of the neural network model to produce subsequent intermediate results until a last layer of the neural network model generates outputs.

### Claim 21 (depends on 20)

The system of claim 20, wherein the set of processing cores and the memory are included within a graphics processing unit.

### Claim 22 (depends on 20)

The system of claim 20, wherein the memory comprises at least one of low-level caches, shared on-chip memory, and registers.

### Claim 23 (depends on 20)

The system of claim 20, wherein the neural network model implements a radiance cache for performing light transport path tracing.

### Claim 24

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
partitioning inputs to a neural network model into portions that are each sized for execution in a processing core by a single thread block;
loading weights for the neural network model into a register file within a processor once to process the inputs;
independently processing the portions in parallel by a set of processing cores within the processor, wherein weights for a first layer of the neural network model are applied to each of the portions to generate intermediate results for each portion;
storing the intermediate results for each portion into a memory that is shared between the set of processing cores; and
processing the intermediate results for each portion by a subsequent layer of the neural network model to produce subsequent intermediate results until a last layer of the neural network model generates outputs.

### Claim 25 (depends on 24)

The non-transitory computer-readable media of claim 24, wherein the processor is a graphics processing unit.

### Claim 26 (depends on 24)

The non-transitory computer-readable media of claim 24, wherein the memory comprises at least one of low-level caches, shared on-chip memory, and registers.

### Claim 27 (depends on 24)

The non-transitory computer-readable media of claim 24, wherein the neural network model implements a radiance cache for performing light transport path tracing.