# US 2018/0293713 A1 · Denoising monte carlo renderings using machine learning with importance sampling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0293713 A1
- Document ID: US20180293713A1
- Status: Published application
- Kind: A1
- Application number: 15/946,654
- Filed: 2018-04-05
- Published: 2018-10-11
- Earliest claimed: 2017-04-06
- Assignees: Pixar
- Applicants: Pixar
- Inventors: Thijs Vogels; Fabrice Rousselle; Brian McWilliams; Mark Meyer; Jan Novak
- Web page: https://patentgrep.com/patent/US20180293713A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/72: Data preparation, e.g. statistical preprocessing of image or video features
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/82: using neural networks
- G06T 7/90: Determination of colour characteristics
- G06T 15/06: Ray-tracing
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/30168: Image quality inspection
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.

## Claims

### Claim 1

A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image;
configuring a neural network comprising:
an input layer configured to receive the set of input images;
a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and
an output layer coupled to a last layer of the plurality of hidden layers and configured to output a respective denoised image corresponding to a respective input image;
training the neural network using the set of input images and the set of reference images, the training comprising:
obtaining one or more image metrics for each respective input image of the set of input images or for a reference image corresponding to the respective input image;
selecting a first input image among the set of input images according to a probability function based on the one or more image metrics;
performing a first iteration of the training using the first input image and a corresponding first reference image to obtain a first intermediate set of parameters associated with the nodes of the plurality of hidden layers;
selecting additional input images among the set of input images according to the probability function; and
performing additional iterations of the training using each of the additional input images and a corresponding reference image to obtain a final set of parameters associated with the nodes of the plurality of hidden layers;
receiving a new input image rendered by MC path tracing; and
generating a new denoised image corresponding to the new input image by passing the new input image through the neural network using the final set of parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein the one or more image metrics relate to one or more of average pixel color variance within the respective input image, variance of surface normals within the respective input image, presence of edges within the respective input image, or variance of effective diffuse irradiance within the respective input image.

### Claim 3 (depends on 2)

The method of claim 2, wherein the set of input images include one or more auxiliary buffers output by a renderer, and wherein the one or more image metrics are obtained from the one or more auxiliary buffers.

### Claim 4 (depends on 2)

The method of claim 2, wherein the one or more image metrics are obtained by analyzing each of the set of input images using an image analysis algorithm.

### Claim 5 (depends on 1)

The method of claim 1, wherein the training of the neural network further comprises:
for each iteration of the training:
passing a respective input image through the neural network to obtain an intermediate denoised image;
comparing the intermediate denoised image to a corresponding reference image to obtain a gradient of a loss function for each pixel; and
back-propagating the gradient of the loss function through the neural network to obtain an updated set of parameters associated with the nodes of the plurality of hidden layers.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
normalizing the gradient of the loss function by the probability function.

### Claim 7 (depends on 6)

The method of claim 6, wherein normalizing the gradient of the loss function comprises dividing the gradient of the loss function by the probability function.

### Claim 8 (depends on 1)

The method of claim 1, wherein each input image of the set of input images is rendered with a first number of samples per pixel, each reference image of the set of reference images is rendered by MC path tracing with a second number of samples per pixel greater than the first number of samples per pixel.

### Claim 9 (depends on 1)

The method of claim 1, wherein the neural network comprises a convolutional neural network.

### Claim 10 (depends on 1)

The method of claim 1, wherein the neural network comprises a multilayer perceptron neural network.

### Claim 11

A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image;
configuring a neural network comprising:
an input layer configured to receive the set of input images;
a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and
an output layer coupled to a last layer of the plurality of hidden layers, the output layer configured to output a respective denoised image corresponding to a respective input image;
training the neural network using the set of input images and the set of reference images, the training comprising:
performing one or more initial iterations of the training by randomly sampling the set of input images using a uniform probability to obtain a set of intermediate denoised images, each intermediate denoised image corresponding to a respective input image;
for each respective input image, evaluating an error gradient by comparing a corresponding intermediate denoised image to the respective input image; and
performing one or more additional iterations of the training by sampling the set of input images according to a probability function based on the error gradient of each input image of the set of input images to obtain a set of optimized parameters associated with the nodes of the plurality of hidden layers;
receiving a new input image rendered by MC path tracing; and
generating a new denoised image corresponding to the new input image by passing the new input image through the neural network using the set of optimized parameters.

### Claim 12 (depends on 11)

The method of claim 11, wherein the probability function is proportional to the error gradient of each input image.

### Claim 13 (depends on 12)

The method of claim 12, further comprising normalizing the error gradient by dividing the error gradient by the probability function.

### Claim 14 (depends on 11)

The method of claim 11, wherein each of the set of input images is rendered with a first number of samples per pixel, each of the set of reference images is rendered by MC path tracing with a second number of samples per pixel greater than the first number of samples per pixel.

### Claim 15 (depends on 11)

The method of claim 11, wherein the neural network comprises a convolutional neural network or a multilayer perceptron neural network.

### Claim 16

A method of denoising images rendered by Monte Carlo (MC) path tracing, the method comprising:
receiving a set of input images rendered by MC path tracing and a set of reference images, each reference image corresponding to a respective input image;
configuring a neural network comprising:
an input layer configured to receive the set of input images;
a plurality of hidden layers, each hidden layer having a respective number of nodes, each node associated with a respective parameter, a first layer of the plurality of hidden layers coupled to the input layer; and
an output layer coupled to a last layer of the plurality of hidden layers, the output layer configured to output a respective denoised image corresponding to a respective input image;
training the neural network using the set of input images and the set of reference images, the training comprising:
assigning a relevance score to each respective input image of the set of input images, the relevance score indicating a degree of relevance to one or more areas of interests; and
performing the training by sampling the set of input images according to a probability function that is proportional to the relevance score of each respective input image to obtain a set of optimized parameters associated with the nodes of the plurality of hidden layers;
receiving a new input image rendered by MC path tracing; and
generating a denoised image corresponding to the new input image by passing the new input image through the neural network using the set of optimized parameters.

### Claim 17 (depends on 16)

The method of claim 16, wherein the one or more areas of interests relate to one or more of presence of hair, presence of a face, or presence of a character in the respective input image.

### Claim 18 (depends on 16)

The method of claim 16, wherein the neural network comprises a convolutional neural network.

### Claim 19 (depends on 16)

The method of claim 16, wherein the neural network comprises a multilayer perceptron neural network.

### Claim 20 (depends on 16)

The method of claim 16, wherein each input image of the set of input images is rendered with a first number of samples per pixel, each reference image of the set of reference images is rendered by MC path tracing with a second number of samples per pixel greater than the first number of samples per pixel.
