# US 12,118,695 B2 · Noise estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,695 B2 - Document ID: US12118695B2 - Status: Granted - Kind: B2 - Application number: 17/480,548 - Filed: 2021-09-21 - Published: 2022-01-06 - Granted: 2024-10-15 - Earliest claimed: 2019-03-21 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Hao Guan; Gregory Slabaugh; Liu Liu; Sean Moran; Zhongqian Fu - Web page: https://patentgrep.com/patent/US12118695B2 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 5/60: using machine learning, e.g. neural networks - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract One example image processing device is provided. The example image processing device can include at least one processor and one or more memories coupled to the at least one processor and storing programming instructions for execution by the at least one processor to estimate noise in an image, the image being represented by a set of pixels and each pixel of the set of pixels having a value associated with it on each of one or more channels, where estimating the noise comprises processing, using a first trained model that detects stochastic noise, data derived from the image to form a first noise estimate, processing, using a second trained model that detects extreme pixel values, data derived from the image to form a second noise estimate, and combining the first and second noise estimates to form an aggregated noise estimate. ## Claims ### Claim 1 An image processing device, comprising: at least one processor; and one or more memories coupled to the at least one processor and storing programming instructions for execution by the at least one processor to estimate noise in an image, the image being represented by a set of pixels, and each pixel of the set of pixels having a value associated with the pixel on each of one or more channels, wherein estimating the noise in the image comprises: processing, using a first trained model that detects stochastic noise, data derived from the image to obtain a first noise estimate, wherein the first trained model detects at least one of Poisson noise or Gaussian noise; processing, using a second trained model that detects extreme pixel values, data derived from the image to obtain a second noise estimate; and combining the first and second noise estimates to obtain an aggregated noise estimate. ### Claim 2 (depends on 1) The image processing device as claimed in claim 1, wherein the first trained model has a higher accuracy in detecting the at least one of Poisson noise or Gaussian noise than the second trained model. ### Claim 3 (depends on 1) The image processing device as claimed in claim 1, wherein the second trained model has a higher accuracy in detecting defective pixel noise than the first trained model. ### Claim 4 (depends on 1) The image processing device as claimed in claim 1, wherein the one or more memories store the programming instructions for execution by the at least one processor to subtract the aggregated noise estimate from the image to obtain a de-noised image. ### Claim 5 (depends on 1) The image processing device as claimed in claim 1, wherein the one or more memories store the programming instructions for execution by the at least one processor to process, using a third trained model, the image and the aggregated noise estimate to obtain a de-noised image. ### Claim 6 (depends on 1) The image processing device as claimed in claim 1, wherein the first and second trained models comprise processing architectures to: (a) process data derived from the image to progressively reduce resolution through a first series of stages to obtain intermediate data; and (b) process the intermediate data to progressively increase resolution through a second series of stages to obtain a respective noise estimate; there being skip connections for providing feed-through between corresponding stages of the first and second series. ### Claim 7 (depends on 6) The image processing device as claimed in claim 6, wherein the first series of stages of the second trained model comprise: (a) a first stage that processes the data derived from the image to reduce the resolution and increase data depth to obtain second intermediate data, and (b) a second stage that processes the second intermediate data to reduce resolution without increasing the data depth to obtain third intermediate data. ### Claim 8 (depends on 7) The image processing device as claimed in claim 7, wherein the first stage is a space-to-depth stage. ### Claim 9 (depends on 7) The image processing device as claimed in claim 7, wherein the second stage is a max pooling stage. ### Claim 10 (depends on 6) The image processing device as claimed in claim 6, wherein none of the stages in the first series of stages of the first trained model increases data depth. ### Claim 11 A method for training an image processing model, comprising: receiving a plurality of pairs of images, wherein each pair of images represents a common scene and a first image of the pair which comprises more noise than a second image of the pair; and for the each pair of images, processing, using a first model that estimates stochastic noise in the images, data derived from the first image of the pair to obtain a first noise estimate; processing, using a second model that detects extreme pixel values, data derived from the first image of the pair to obtain a second noise estimate; combining the first noise estimate and the second noise estimate to obtain an aggregated noise estimate; estimating a difference between the second image of the pair and the first image of the pair de-noised in dependence on the aggregated noise estimate; and adapting the first and second models in dependence on the estimated difference. ### Claim 12 (depends on 11) The method as claimed in claim 11, wherein: the first and second models comprise processing architectures to: (a) process data derived from the first image of the pair to progressively reduce resolution through a first series of stages to obtain intermediate data; and (b) process the intermediate data to progressively increase resolution through a second series of stages to obtain a respective noise estimate; there being skip connections for providing feed-through between corresponding stages of the first and second series; and wherein the first series of stages of the second model comprise: (a) a first stage that processes the data derived from the first image of the pair to reduce the resolution and increase data depth to obtain second intermediate data, and (b) a second stage that processes the second intermediate data to reduce resolution without increasing data depth to obtain third intermediate data. ### Claim 13 (depends on 11) An image processing model adapted by the method of claim 11. ### Claim 14 An image processing device comprising at least one processor and one or more memories, the one or more memories storing in non-transient form instructions executable by the at least one processor to implement an image processing model to: receive a plurality of pairs of images, wherein each pair of images represents a common scene and a first image of the pair which comprises more noise than a second image of the pair; and for the each pair of images, process, using a first model that estimates stochastic noise in the images, data derived from the first image of the pair to obtain a first noise estimate; process, using a second model that detects extreme pixel values, data derived from the first image of the pair to obtain a second noise estimate; combine the first noise estimate and the second noise estimate to obtain an aggregated noise estimate; estimate a difference between the second image of the pair and the first image of the pair de-noised in dependence on the aggregated noise estimate; and adapt the first and second models in dependence on the estimated difference. ### Claim 15 (depends on 14) The image processing device as claimed in claim 14, wherein: the first and second models comprise processing architectures to: (a) process data derived from the first image of the pair to progressively reduce resolution through a first series of stages to obtain intermediate data; and (b) process the intermediate data to progressively increase resolution through a second series of stages to obtain a respective noise estimate; there being skip connections for providing feed-through between corresponding stages of the first and second series; and wherein the first series of stages of the second model comprise: (a) a first stage that processes the data derived from the first image of the pair to reduce the resolution and increase data depth to obtain second intermediate data, and (b) a second stage that processes the second intermediate data to reduce resolution without increasing data depth to obtain third intermediate data.