# US 11,113,800 B2 · Filtering image data using a neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,113,800 B2 - Document ID: US11113800B2 - Status: Granted - Kind: B2 - Application number: 15/872,838 - Filed: 2018-01-16 - Published: 2018-07-19 - Granted: 2021-09-07 - Earliest claimed: 2017-01-18 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Anton S. Kaplanyan; Chakravarty Reddy Alla Chaitanya; Timo Oskari Aila; Aaron Eliot Lefohn; Marco Salvi - Web page: https://patentgrep.com/patent/US11113800B2 ## CPC - G06V 10/82: using neural networks - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 5/20: using local operators - G06T 5/60: using machine learning, e.g. neural networks - G06T 5/70: Denoising; Smoothing - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/955: {using specific electronic processors} - G06V 30/19173: {Classification techniques} - G06V 30/194: References adjustable by an adaptive method, e.g. learning - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes identifying image data to be rendered, reconstructing the image data to create reconstructed image data, utilizing a filter including a neural network having one or more skip connections and one or more recurrent layers, and returning the reconstructed image data. ## Claims ### Claim 1 A method for performing spatiotemporal filtering, comprising: identifying image data to be rendered, the image data including: color data, and one or more auxiliary features that include at least one of one or more view-space shading normals, one or more depth values, or one or more roughness values; reconstructing the image data to create reconstructed image data, utilizing a filter including a neural network that processes the image data based on the one or more auxiliary features, the neural network having: one or more skip connections, and one or more recurrent layers that retain, at least in part, the one or more auxiliary features unchanged; and returning the reconstructed image data and the unchanged one or more auxiliary features. ### Claim 2 (depends on 1) The method of claim 1, wherein the image data is created as a result of a predetermined process, the predetermined process including one or more of: low light photography; low light video recording; a Monte Carlo sampling method; photography that produces RGB and depth information; and depth-only cameras. ### Claim 3 (depends on 1) The method of claim 1, wherein the image data has a number of samples per pixel (spp) below a predetermined threshold. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more auxiliary features includes the one or more view-space shading normals. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more auxiliary features includes the one or more depth values. ### Claim 6 (depends on 1) The method of claim 1, wherein the one or more auxiliary features includes the one or more roughness values. ### Claim 7 (depends on 1) The method of claim 1, wherein the neural network includes a plurality of convolution layers. ### Claim 8 (depends on 1) The method of claim 1, wherein the neural network includes a plurality of encoder layers and corresponding decoder layers, and a first skip connection within the neural network enables a reconstruction process to jump from a first encoder layer directly to a corresponding decoder layer, thereby skipping over one or more encoder and decoder layers within the neural network. ### Claim 9 (depends on 1) The method of claim 1, wherein the one or more recurrent layers include a recurrent convolutional neural network (RCNN). ### Claim 10 (depends on 1) The method of claim 1, wherein the one or more recurrent layers link a layer to itself within the neural network. ### Claim 11 (depends on 1) The method of claim 1, further comprising training the neural network, utilizing a data set including a plurality of input/output pairs. ### Claim 12 (depends on 11) The method of claim 11, wherein a loss function is utilized during the training of the neural network. ### Claim 13 (depends on 1) The method of claim 1, wherein the image data includes a first frame of a sequence of frames, and wherein the one or more auxiliary features used during the reconstructing of the first frame is stored as a hidden, recurrent state, which is then accessed during a reconstruction of a second frame of the sequence of frames of the image data within an RCNN, utilizing a feedback loop of the RCNN. ### Claim 14 A system comprising: a processor that is configured to: identify image data to be rendered, the image data including: color data, and one or more auxiliary features that include at least one of one or more view-space shading normals, one or more depth values, or one or more roughness values; reconstruct the image data to create reconstructed image data, utilizing a filter including a neural network that processes the image data based on the one or more auxiliary features, the neural network having: one or more skip connections, and one or more recurrent layers that retain, at least in part, the one or more auxiliary features unchanged; and return the reconstructed image data and the unchanged one or more auxiliary features. ### Claim 15 (depends on 14) The system of claim 14, wherein the image data is created as a result of a predetermined process, the predetermined process including one or more of: low light photography; low light video recording; a Monte Carlo sampling method; photography that produces RGB and depth information; and depth-only cameras. ### Claim 16 (depends on 14) The system of claim 14, wherein the image data has a number of samples per pixel (spp) below a predetermined threshold. ### Claim 17 (depends on 14) The system of claim 14, wherein the one or more auxiliary features includes the one or more view-space shading normals. ### Claim 18 A method for performing spatiotemporal filtering, comprising: identifying image data to be rendered, the image data resulting from a Monte Carlo sampling method and the image data including: color data, and one or more auxiliary features that include at least one of one or more view-space shading normals, one or more depth values, or one or more roughness values; applying a filter to the image data to create reconstructed image data, where the filter is a neural network that processes the image data based on the one or more auxiliary features, the neural network having: one or more skip connections, and one or more recurrent layers that retain, at least in part, the one or more auxiliary features unchanged; and returning the reconstructed image data and the unchanged one or more auxiliary features. ### Claim 19 (depends on 1) The method of claim 1, wherein the image data and the one or more auxiliary features are identified from a deep image consisting of multiple buffers storing the image data and the one or more auxiliary features. ### Claim 20 (depends on 11) The method of claim 11, wherein the neural network is trained by: demodulating an input image of an input and output pair to remove texture from the input image; processing the demodulated input image to reconstruct the demodulated input image; and generating a final rendering by re-modulating the reconstructed demodulated input image to include the texture.