# US 2022/0284657 A1 · Real-time neural network radiance caching for path tracing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284657 A1 - Document ID: US20220284657A1 - Status: Published application - Kind: A1 - Application number: 17/340,222 - Filed: 2021-06-07 - Published: 2022-09-08 - Earliest claimed: 2021-03-03 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Thomas Müller; Fabrice Pierre Armand Rousselle; Jan Novák; Alexander Georg Keller - Web page: https://patentgrep.com/patent/US20220284657A1 ## CPC - G06T 15/06: Ray-tracing - G06N 3/0499: Feedforward networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06T 15/506: {Illumination models} - G06N 3/045: Combinations of networks - G06N 3/048: Activation functions - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A real-time neural radiance caching technique for path-traced global illumination is implemented using a neural network for caching scattered radiance components of global illumination. The neural (network) radiance cache handles fully dynamic scenes, and makes no assumptions about the camera, lighting, geometry, and materials. In contrast with conventional caching, the data-driven approach sidesteps many difficulties of caching algorithms, such as locating, interpolating, and updating cache points. The neural radiance cache is trained via online learning during rendering. Advantages of the neural radiance cache are noise reduction and real-time performance. Importantly, the runtime overhead and memory footprint of the neural radiance cache are stable and independent of scene complexity. ## Claims ### Claim 1 A computer-implemented method, comprising: processing, by a neural network radiance cache model, a three-dimensional (3D) position associated with a light transport path through a scene to produce a radiance prediction at the 3D position; generating an image of the scene corresponding to at least one camera using the radiance prediction; generating an extended light transport path ending at a terminal 3D position; processing the terminal 3D position to produce a second radiance prediction at the terminal 3D position; and updating weights of the neural network radiance cache model based on the second radiance prediction. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the extended light transport path is generated by extending the light transport path from the 3D position. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the updated weights are also updated based on the second radiance prediction propagated to the 3D position. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the extended light transport path does not include the 3D position. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the terminal 3D position is processed by the neural network radiance cache model to produce the second radiance prediction. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the light transport path terminates at the 3D position based on a first size of a first footprint of the light transport path at the 3D position compared with a second size of a second footprint of the light transport path at a visible 3D position. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the extended light transport path terminates at the terminal 3D position based on a third size of a third footprint of the extended light transport path at the terminal 3D position compared with the second size. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the neural network radiance cache model also processes an additional input that is at least one of a material property, view direction, or normal vector associated with the 3D position to produce the radiance prediction. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein, prior to processing the 3D position and the additional input, the 3D position and the additional input are encoded into a higher-dimensional space. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein a filtered copy of the weights is used to generate the radiance prediction at the 3D position. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, wherein updating the weights produces a sequence of weight updates and the filter is a moving average over the sequence of weight updates. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the light transport path and additional light transport paths are traced to produce one light transport path for each pixel of the image, and further comprising selecting a subset including the additional light transport paths and extending each additional light transport path in the subset to produce additional extended light transport paths. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein the subset is chosen according to at least one of a uniform and a non-uniform distribution. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, further comprising: processing, by the neural network radiance cache model, a second 3D position associated with a second light transport path through the scene according to the updated weights to produce a second radiance prediction at the second 3D position; and generating a second image of the scene using the second radiance prediction. ### Claim 15 (depends on 14) The computer-implemented method of claim 14, wherein at least one of the camera, lighting, geometry, and materials is changed for the second light transport path compared with the light transport path. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein the radiance prediction is multiplied by at least one of diffuse reflectance, specular reflectance, a normal vector, roughness, albedo, bidirectional reflectance distribution function (BRDF), and a phase function of a material property associated with the 3D position. ### Claim 17 (depends on 1) The computer-implemented method of claim 1, wherein a second 3D position at which a second radiance prediction is produced is importance sampled based on the 3D position. ### Claim 18 (depends on 1) The computer-implemented method of claim 1, wherein the radiance prediction comprises only ambient occlusion. ### Claim 19 (depends on 1) The computer-implemented method of claim 1, wherein the neural network radiance cache model processes additional 3D positions associated with additional light transport paths from additional cameras. ### Claim 20 (depends on 1) The computer-implemented method of claim 1, wherein the 3D position is generated by at least one of a rasterizer, a ray tracer, a ray marcher, a sphere tracer, and a cone tracer. ### Claim 21 (depends on 1) The computer-implemented method of claim 1, wherein the 3D position is parameterized by 2D surface coordinates. ### Claim 22 (depends on 1) The computer-implemented method of claim 1, wherein the neural network radiance cache model is trained on a server or in a data center and the image is streamed to a user device. ### Claim 23 (depends on 1) The computer-implemented method of claim 1, wherein the neural network radiance cache model is trained on a server or in a data center and is streamed to a user device. ### Claim 24 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the steps of processing the 3D position, generating the image, generating the extended light transport path, processing the terminal 3D position, and updating are performed within a cloud computing environment. ### Claim 25 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the steps of processing the 3D position, generating the image, generating the extended light transport path, processing the terminal 3D position, and updating are performed on a server or in a data center and the image is streamed to a user device. ### Claim 26 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the steps of processing the 3D position, generating the image, generating the extended light transport path, processing the terminal 3D position, and updating are performed on a server or in a data center and the neural network radiance cache is streamed to a user device. ### Claim 27 (depends on 1) The computer-implemented method of claim 1, wherein the generated image is used for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle. ### Claim 28 (depends on 1) The computer-implemented method of claim 1, wherein one or more of the steps of processing the 3D position, generating the image, generating the extended light transport path, processing the terminal 3D position, and updating are performed on a virtual machine comprising a portion of a graphics processing unit. ### Claim 29 A system, comprising: a memory storing data for a scene; a processor that is connected to the memory and configured to generate images of the scene by: processing, by a neural network radiance cache model, a three-dimensional (3D) position associated with a light transport path through a scene to produce a radiance prediction at the 3D position; generating an image of the scene corresponding to at least one camera using the radiance prediction; generating an extended light transport path ending at a terminal 3D position; processing the terminal 3D position to produce a second radiance prediction at the terminal 3D position; and updating weights of the neural network radiance cache model based on the second radiance prediction. ### Claim 30 (depends on 29) The system of claim 29, further comprising terminating the light transport path at the 3D position based on a first size of a first footprint of the light transport path at the 3D position compared with a second size of a second footprint of the light transport path at a visible 3D position. ### Claim 31 (depends on 29) The system of claim 29, wherein the neural network radiance cache model also processes an additional input that is at least one of a material property, view direction, or normal vector associated with the 3D position to produce the radiance prediction. ### Claim 32 (depends on 29) The system of claim 29, wherein a filtered copy of the weights is used to generate the radiance prediction at the 3D position. ### Claim 33 (depends on 29) The system of claim 29, the processing comprises partitioning the 3D position and additional 3D positions into portions that are each sized for execution in a processing core by a single thread block; and independently processing the portions in parallel by a set of processing cores within the processor, wherein the weights are applied to each of the portions to produce the radiance prediction at the 3D position and additional radiance predictions at the additional 3D positions. ### Claim 34 A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: processing, by a neural network radiance cache model, a three-dimensional (3D) position associated with a light transport path through a scene to produce a radiance prediction at the 3D position; generating an image of the scene corresponding to at least one camera using the radiance prediction; generating an extended light transport path ending at a terminal 3D position; processing the terminal 3D position to produce a second reflected radiance at the terminal 3D position; and updating weights of the neural network radiance cache model based on the second radiance prediction. ### Claim 35 (depends on 34) The non-transitory computer-readable media of claim 34, wherein the light transport path terminates at the 3D position based on a first size of a first footprint of the light transport path at the 3D position compared with a second size of a second footprint of the light transport path at a visible 3D position.