# US 12,602,738 B2 · Noise reduction circuit with demosaic processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,738 B2 - Document ID: US12602738B2 - Status: Granted - Kind: B2 - Application number: 18/128,479 - Filed: 2023-03-30 - Published: 2024-10-03 - Granted: 2026-04-14 - Earliest claimed: 2023-03-30 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Maxim Smirnov - Web page: https://patentgrep.com/patent/US12602738B2 ## CPC - G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns - G06T 5/20: using local operators - G06T 5/70: Denoising; Smoothing - G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A noise reduction circuit that performs demosaic-based noise reduction on image data. The noise reduction circuit includes a pre-demosaic circuit, a kernel calculation circuit, a noise filtering circuit, and a blending circuit. The pre-demosaic circuit generates a de-mosaiced version of an image. The kernel calculation circuit generates a denoising kernel for at least one pixel of the image and a bilateral kernel for the at least one pixel of the image using the de-mosaiced version of the image. The noise filtering circuit performs noise filtering of the image using the denoising kernel to generate a first de-noised version of the image, and performs noise filtering of the image using the bilateral kernel to generate a second de-noised version of the image. The blending circuit blends the first de-noised version with the second de-noised version to generate an output de-noised version of the image. ## Claims ### Claim 1 A circuit, comprising: a pre-demosaic circuit to generate a de-mosaiced version of an image; a kernel calculation circuit coupled to the pre-demosaic circuit, the kernel calculation circuit to generate, using the de-mosaiced version of the image, a denoising kernel for at least one pixel of the image and a bilateral kernel for the at least one pixel of the image; a noise filtering circuit coupled to the kernel calculation circuit and the pre-demosaic circuit, the noise filtering circuit to: perform noise filtering of the image using the denoising kernel to generate a first de-noised version of the image, and perform noise filtering of the image using the bilateral kernel to generate a second de-noised version of the image; and a blending circuit coupled to the kernel calculation circuit and the noise filtering circuit, the blending circuit to blend the first de-noised version of the image with the second de-noised version of the image to generate an output de-noised version of the image. ### Claim 2 (depends on 1) The circuit of claim 1, wherein the kernel calculation circuit is to: classify a corresponding patch of the image using a plurality of analysis kernels to obtain a plurality of classification weights for the at least one pixel of the image; calculate absolute values of the plurality of classification weights to generate a vector of absolute weights for the at least one pixel of the image; normalize the vector of absolute weights by a patch standard deviation for the at least one pixel of the image to obtain a vector of normalized weights for the at least one pixel of the image; calculate a quality factor for the at least one pixel of the image using the vector of normalized weights, the quality factor representing a quality measure for the denoising kernel for the at least one pixel of the image; and compute, for the at least one pixel of the image, an alpha parameter by inputting the quality factor into a look-up table (LUT) circuit, and wherein the blending circuit is to blend the first de-noised version of the image with the second de-noised version of the image using the alpha parameter to generate the output de-noised version of the image. ### Claim 3 (depends on 2) The circuit of claim 2, wherein the kernel calculation circuit is to determine the patch standard deviation for the at least one pixel of the image by calculating a standard deviation between the at least one pixel of the image and other pixels in the corresponding patch of the image. ### Claim 4 (depends on 1) The circuit of claim 1, wherein the kernel calculation circuit is further configured to: classify a corresponding patch of the image using a plurality of analysis kernels to obtain a plurality of classification weights for the at least one pixel of the image; process the classification weights for the at least one pixel of the image to generate a plurality of normalized weights for the at least one pixel of the image; calculate a weighted average of a plurality of synthesis kernels using the plurality of normalized weights to generate a version of the denoising kernel for the at least one pixel of the image; and adjust the version of the denoising kernel using Mahalanobis distances between at least one pixel in the de-mosaiced version of the image and other pixels in the corresponding patch of the image to generate the denoising kernel for the at least one pixel of the image. ### Claim 5 (depends on 4) The circuit of claim 4, wherein the kernel calculation circuit is to: calculate absolute values of the plurality of classification weights to generate a vector of absolute weights for the at least one pixel of the image; normalize the vector of absolute weights by a patch standard deviation for the at least one pixel of the image to obtain a vector of normalized weights for the at least one pixel of the image; and process the vector of normalized weights by inputting the vector of normalized weights into a look-up table (LUT) circuit to generate the plurality of normalized weights for the at least one pixel of the image. ### Claim 6 (depends on 1) The circuit of claim 1, wherein the kernel calculation circuit is to generate the denoising kernel by processing a corresponding patch of the image using at least one filter bank. ### Claim 7 (depends on 1) The circuit of claim 1, wherein the kernel calculation circuit is to: generate an initial kernel for the at least one pixel of the image by processing a corresponding patch of the image using at least one filter bank; generate the denoising kernel by adjusting the initial kernel using photometric distances between at least one pixel in the de-mosaiced version of the image and pixels in the corresponding patch of the image; and generate the bilateral kernel by computing Mahalanobis distances between the at least one pixel in the de-mosaiced version of the image and other pixels in a corresponding patch of the image. ### Claim 8 (depends on 1) The circuit of claim 1, wherein the image comprises an unscaled single color version of an input image, and the output de-noised version of the image comprises a de-noised unscaled single color version of the input image. ### Claim 9 A method, comprising: generating a de-mosaiced version of an image; generating, using the de-mosaiced version of the image, a denoising kernel for at least one pixel of the image and a bilateral kernel for the at least one pixel of the image; performing noise filtering of the image using the denoising kernel to generate a first de-noised version of the image; performing noise filtering of the image using the bilateral kernel to generate a second de-noised version of the image; and blending the first de-noised version with the second de-noised version to generate an output de-noised version of the image. ### Claim 10 (depends on 9) The method of claim 9, further comprising: classifying a corresponding patch of the image using a plurality of analysis kernels to obtain a plurality of classification weights for the at least one pixel of the image; calculating, absolute values of the plurality of classification weights to generate a vector of absolute weights for the at least one pixel of the image; determining a patch standard deviation for the at least one pixel of the image by calculating a standard deviation between the at least one pixel of the image and other pixels in the corresponding patch of the image; normalizing the vector of absolute weights by the patch standard deviation for the at least one pixel of the image to obtain a vector of normalized weights for the at least one pixel of the image; calculating a quality factor for the at least one pixel of the image using the vector of normalized weights, the quality factor representing a quality measure for the denoising kernel for the at least one pixel of the image; computing, for the at least one pixel of the image, an alpha parameter by inputting the quality factor into a look-up table (LUT) circuit; and blending the first de-noised version of the image with the second de-noised version of the image using the alpha parameter. ### Claim 11 (depends on 9) The method of claim 9, further comprising: classifying a corresponding patch of the image using a plurality of analysis kernels to obtain a plurality of classification weights for the at least one pixel of the image; processing the classification weights for the at least one pixel of the image to generate a plurality of normalized weights for the at least one pixel of the image; calculating a weighted average of a plurality of synthesis kernels using the plurality of normalized weights to generate a version of the denoising kernel for the at least one pixel of the image; and adjusting the version of the denoising kernel using Mahalanobis distances between at least one pixel in the de-mosaiced version of the image and other pixels in the corresponding patch of the image to generate the denoising kernel for the at least one pixel of the image. ### Claim 12 (depends on 11) The method of claim 11, further comprising: calculating absolute values of the plurality of classification weights to generate a vector of absolute weights for the at least one pixel of the image; normalizing the vector of absolute weights for the at least one pixel of the image by a patch standard deviation for the at least one pixel of the image to obtain a vector of normalized weights for each pixel of the image; and processing the vector of normalized weights for the at least one pixel of the image by inputting the vector of normalized weights into a look-up table (LUT) circuit to generate the plurality of normalized weights for the at least one pixel of the image. ### Claim 13 (depends on 9) The method of claim 9, further comprising: generating an initial kernel for the at least one pixel of the image by processing a corresponding patch of the image using at least one filter bank; generating the denoising kernel by adjusting the initial kernel using photometric distances between at least one pixel in the de-mosaiced version of the image and pixels in the corresponding patch of the image; and generating the bilateral kernel by computing Mahalanobis distances between the at least one pixel in the de-mosaiced version of the image and other pixels in the corresponding patch of the image. ### Claim 14 An electronic device, comprising: an image signal processor to perform noise reduction on an image, the image signal processor including: a machine learning-based noise reduction (MLNR) circuit, the MLNR circuit comprising: a pre-demosaic circuit to generate a de-mosaiced version of the image; a kernel calculation circuit coupled to the pre-demosaic circuit and to generate, using the de-mosaiced version of the image, a denoising kernel for at least one pixel of an image and a bilateral kernel for the at least one pixel of the image, a noise filtering circuit coupled to the kernel calculation circuit, the noise filtering circuit to: perform noise filtering of the image using the denoising kernel to generate a first de-noised version of the image, and perform noise filtering of the image using the bilateral kernel to generate a second de-noised version of the image, and a blending circuit coupled to the kernel calculation circuit and the noise filtering circuit, the blending circuit to blend the first de-noised version with the second de-noised version to generate an output de-noised version of the image. ### Claim 15 (depends on 1) The circuit of claim 1, wherein the pre-demosaic circuit is to estimate each color component for each pixel of the image to generate the de-mosaiced version of the image. ### Claim 16 (depends on 1) The circuit of claim 1, wherein the noise filtering circuit comprises: a noise filtering and de-mosaicing circuit to perform noise filtering and de-mosaicing of the image using the denoising kernel and demosaic processing to generate the first de-noised version of the image; and a bilateral noise filtering circuit to perform noise filtering of the image using the bilateral kernel and demosaic processing to generate the second de-noised version of the image. ### Claim 17 (depends on 1) The circuit of claim 1, further comprising a combining circuit coupled to the blending circuit to combine the image with the output de-noised version of the image to generate a final de-noised version of the image. ### Claim 18 (depends on 9) The method of claim 9, further comprising: estimating each color component for each pixel of the image to generate the de-mosaiced version of the image. ### Claim 19 (depends on 9) The method of claim 9, further comprising: combining the image with the output de-noised version of the image to generate a final de-noised version of the image. ### Claim 20 (depends on 14) The electronic device of claim 14, wherein the kernel calculation circuit is to: generate an initial kernel for the at least one pixel of the image by processing a corresponding patch of the image using at least one filter bank; generate the denoising kernel by adjusting the initial kernel using photometric distances between at least one pixel in the de-mosaiced version of the image and pixels in the corresponding patch of the image; and generate the bilateral kernel for the at least one pixel of the image by computing Mahalanobis distances between the at least one pixel in the de-mosaiced version of the image and other pixels in the corresponding patch of the image.