# US 10,311,552 B2 · De-noising images using machine learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,552 B2 - Document ID: US10311552B2 - Status: Granted - Kind: B2 - Application number: 15/630,478 - Filed: 2017-06-22 - Published: 2018-10-11 - Granted: 2019-06-04 - Earliest claimed: 2017-04-06 - Assignees: Pixar - Applicants: Pixar - Inventors: Mark Meyer; Anthony DeRose; Steve Bako - Web page: https://patentgrep.com/patent/US10311552B2 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 5/60: using machine learning, e.g. neural networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure relates to using a neural network to efficiently denoise images that were generated by a ray tracer. The neural network can be trained using noisy images generated with noisy samples and corresponding denoised or high-sampled images (e.g., many random samples). An input feature to the neural network can include color from pixels of an image. Other input features to the neural network, which would not be known in normal image processing, can include shading normal, depth, albedo, and other characteristics available from a computer-generated scene. After the neural network is trained, a noisy image that the neural network has not seen before can have noise removed without needing manual intervention. ## Claims ### Claim 1 A method for reducing noise in a two-dimensional image obtained from a computer-generated scene, the method comprising: receiving a plurality of noisy subimages, wherein a subimage is at least a portion of an image, and wherein each of the plurality of noisy subimages is rendered using a respective set of rays in a corresponding computer-generated scene; receiving a plurality of refined subimages, each refined subimage corresponding to a noisy subimage of the plurality of noisy subimages and generated using a corresponding computer-generated scene, wherein a refined subimage has reduced noise relative to a corresponding noisy subimage; extracting values of a first set of features from pixels in each of the plurality of noisy subimages; determining values of a second set of features from information obtained from the corresponding computer-generated scenes from which the plurality of noisy subimages were generated; training a neural network based on the values of the features and the plurality of refined subimages, wherein the neural network outputs pixel values of an output subimage or outputs gradients of pixels of the output subimage, and wherein the training includes: generating, using the neural network, an output subimage for a noisy subimage of the plurality of noisy subimages; comparing the output subimage to a refined subimage of the plurality of refined subimages that corresponds to the noisy subimage; and modifying one or more parameters of the neural network based on the comparison; receiving a new two-dimensional image rendered from a computer-generated scene; and generating, using the neural network after training, a new output image corresponding to the new two-dimensional image, wherein the neural network is a first neural network, wherein the first set of features includes an illumination of pixels of the noisy subimage, wherein the first neural network is trained based on a first light path, wherein a second neural network is trained based on a second light path, and wherein output of the first neural network and the second neural network are combined to generate the new output image. ### Claim 2 (depends on 1) The method of claim 1, wherein a first noisy subimage of the plurality of noisy subimages is rendered using a respective set of rays corresponding to a first computer-generated scene, and wherein a second noisy subimage of the plurality of noisy subimages is rendered using a respective different set of rays for a different set of pixels corresponding to the first computer-generated scene. ### Claim 3 (depends on 1) The method of claim 1, wherein the refined subimage is generated using more rays than the corresponding noisy subimage. ### Claim 4 (depends on 1) The method of claim 1, wherein the refined subimage is a processed version of the corresponding noisy subimage, and wherein the processing includes removing at least a portion of noise from the corresponding noisy subimage using a filter. ### Claim 5 (depends on 1) The method of claim 1, wherein one of the first set of features is associated with a color or an illumination of each pixel of the noisy subimage. ### Claim 6 (depends on 1) The method of claim 1, wherein the first light path is associated with specular light transport of the pixels of the noisy subimage, and wherein the second light path is associated with diffuse light transport of the pixels of the noisy subimage. ### Claim 7 (depends on 6) The method of claim 6, wherein the diffuse light transport of the pixels is divided by an albedo of the pixels before being trained using the second neural network. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more parameters that are modified include one or more weights that were determined by the neural network, wherein comparing the output subimage to the refined subimage that corresponds to the noisy subimage includes calculating an error between pixel values of the output subimage and pixel values of the refined subimage. ### Claim 9 A non-transitory computer-readable medium storing a computer program that when executed by a processor of a computer cause the processor to implement a computer-implemented method comprising: receiving a plurality of noisy subimages, wherein a subimage is at least a portion of an image, and wherein each of the plurality of noisy subimages is rendered using a respective set of rays in a corresponding computer-generated scene; receiving a plurality of refined subimages, each refined subimage corresponding to a noisy subimage of the plurality of noisy subimages and generated using a corresponding computer-generated scene, wherein a refined subimage has reduced noise relative to a corresponding noisy subimage; extracting values of a first set of features from pixels in each of the plurality of noisy subimages; determining values of a second set of features from information obtained from the corresponding computer-generated scenes from which the plurality of noisy subimages were generated; training a neural network based on the values of the features and the plurality of refined subimages, wherein the neural network outputs pixel values of an output subimage or outputs gradients of pixels of the output subimage, and wherein the training includes: generating, using the neural network, an output subimage for a noisy subimage of the plurality of noisy subimages; comparing the output subimage to a refined subimage of the plurality of refined subimages that corresponds to the noisy subimage; and modifying one or more parameters of the neural network based on the comparison; receiving a new two-dimensional image rendered from a computer-generated scene; and generating, using the neural network after training, a new output image corresponding to the new two-dimensional image, wherein the neural network is a first neural network, wherein the first set of features includes an illumination of pixels of the noisy subimage, wherein the first neural network is trained based on a first light path, wherein a second neural network is trained based on a second light path, and wherein output of the first neural network and the second neural network are combined to generate the new output image. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein a first noisy subimage of the plurality of noisy subimages is rendered using a respective set of rays corresponding to a first computer-generated scene, and wherein a second noisy subimage of the plurality of noisy subimages is rendered using a respective different set of rays for a different set of pixels corresponding to the first computer-generated scene. ### Claim 11 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the refined subimage is generated using more rays than the corresponding noisy subimage. ### Claim 12 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the refined subimage is a processed version of the corresponding noisy subimage, and wherein the processing includes removing at least a portion of noise from the corresponding noisy subimage using a filter. ### Claim 13 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein one of the first set of features is associated with a color or an illumination of each pixel of the noisy subimage. ### Claim 14 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the one or more parameters that are modified include one or more weights that were determined by the neural network, wherein comparing the output subimage to the refined subimage that corresponds to the noisy subimage includes calculating an error between pixel values of the output subimage and pixel values of the refined subimage. ### Claim 15 A method for reducing noise in a two-dimensional image obtained from a computer-generated scene, the method comprising: receiving a plurality of noisy subimages, wherein a subimage is at least a portion of an image, and wherein each of the plurality of noisy subimages is rendered using a respective set of rays in a corresponding computer-generated scene; receiving a plurality of refined subimages, each refined subimage corresponding to a noisy subimage of the plurality of noisy subimages and generated using a corresponding computer-generated scene, wherein a refined subimage has reduced noise relative to a corresponding noisy subimage; extracting values of a first set of features from pixels in each of the plurality of noisy subimages; determining values of a second set of features from information obtained from the corresponding computer-generated scenes from which the plurality of noisy subimages were generated; training a neural network based on the values of the features and the plurality of refined subimages, wherein the neural network outputs pixel values of an output subimage or outputs gradients of pixels of the output subimage, and wherein the training includes: generating, using the neural network, an output subimage for a noisy subimage of the plurality of noisy subimages; comparing the output subimage to a refined subimage of the plurality of refined subimages that corresponds to the noisy subimage; and modifying one or more parameters of the neural network based on the comparison; receiving a new two-dimensional image rendered from a computer-generated scene; and generating, using the neural network after training, a new output image corresponding to the new two-dimensional image, wherein one of the first set of features is a variance of values of different rays for a pixel. ### Claim 16 A computing system comprising: one or more processors configured to: receive a plurality of noisy subimages, wherein a subimage is at least a portion of an image, and wherein each of the plurality of noisy subimages is rendered using a respective set of rays in a corresponding computer-generated scene; receive a plurality of refined subimages, each refined subimage corresponding to a noisy subimage of the plurality of noisy subimages and generated using a corresponding computer-generated scene, wherein a refined subimage has reduced noise relative to a corresponding noisy subimage; extract values of a first set of features from pixels in each of the plurality of noisy subimages; determine values of a second set of features from information obtained from the corresponding computer-generated scenes from which the plurality of noisy subimages were generated; train a neural network based on the values of the features and the plurality of refined subimages, wherein the neural network outputs pixel values of an output subimage or outputs gradients of pixels of the output subimage, and wherein the training includes: generating, using the neural network, an output subimage for a noisy subimage of the plurality of noisy subimages; comparing the output subimage to a refined subimage of the plurality of refined subimages that corresponds to the noisy subimage; and modifying one or more parameters of the neural network based on the comparison; receive a new two-dimensional image rendered from a computer-generated scene; and generate, using the neural network after training, a new output image corresponding to the new two-dimensional image, wherein one of the first set of features is a variance of values of different rays for a pixel. ### Claim 17 (depends on 16) The computing system of claim 16, wherein one of the first set of features is associated with a color or an illumination of each pixel of the noisy subimage, and wherein one of the second set of features is associated with a shading normal, a depth, or an albedo used to generate the noisy subimage. ### Claim 18 (depends on 16) The computing system of claim 16, wherein the neural network is a first neural network, wherein the first set of features includes an illumination of pixels of the noisy subimage, wherein the first neural network is trained based on a first light path, wherein a second neural network is trained based on a second light path, and wherein output of the first neural network and the second neural network are combined to generate the new output image.