# US 2019/0378246 A1 · Method, apparatus, and system for multilevel bilateral noise filtering > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378246 A1 - Document ID: US20190378246A1 - Status: Published application - Kind: A1 - Application number: 16/006,605 - Filed: 2018-06-12 - Published: 2019-12-12 - Earliest claimed: 2018-06-12 - Assignees: Carl Zeiss AG - Applicants: Carl Zeiss AG - Inventors: Wayne E. Prentice; Tanja Teuber; David Abreu - Web page: https://patentgrep.com/patent/US20190378246A1 ## CPC - G06T 5/70: Denoising; Smoothing - G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns - G06T 5/20: using local operators - G06T 2207/10024: Color image - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20028: Bilateral filtering - H04N 2209/045: using mosaic colour filter - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method, an image processing apparatus, and computer program product for multilevel bilateral noise filtering of digital images are provided. The method includes receiving from an image sensor an image frame including a color mosaic having a color filter array (CFA) pattern, decomposing the image frame into at least one color plane image for each of a plurality of color planes, sequentially and separately reducing noise in the at least one color plane image by performing multilevel bilateral noise filtering of the at least one color plane image, and reconstructing the at least one color plane image to generate a noise filtered image frame. ## Claims ### Claim 1 A method for multilevel bilateral noise filtering of digital images, the method comprising: receiving from an image sensor an image frame including a color mosaic having a color filter array (CFA) pattern; decomposing the image frame into at least one color plane image for each of a plurality of color planes; sequentially and separately reducing noise in the at least one color plane image by performing multilevel bilateral noise filtering of the at least one color plane image; and reconstructing the at least one color plane image to generate a noise filtered image frame. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of color planes includes at least one of a green color plane, a red color plane, and a blue color plane. ### Claim 3 (depends on 1) The method of claim 1, wherein performing the multilevel bilateral noise filtering comprises: defining a plurality of noise cleaning levels; bilaterally filtering the at least one color plane image in each of the plurality of noise cleaning levels; and performing frequency splitting of the at least one color plane image in each of the plurality of noise cleaning levels except for a highest noise cleaning level. ### Claim 4 (depends on 3) The method of claim 3, wherein frequency splitting comprises: in a first noise cleaning level, performing Gaussian filtering of the at least one color plane image, dividing the at least one color plane image into low and high frequency color plane images, the low frequency color plane image being generated by downsizing the Gaussian filtered at least one color plane image, and the high frequency color plane image being generated by upsizing the low frequency color plane image and subtracting respective pixel values of the upsized low frequency color plane from corresponding pixel values of the at least one color plane image, and forwarding the low frequency color plane image to a subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level; and in at least one noise cleaning level subsequent to the first noise cleaning level and preceding the highest noise cleaning level, performing Gaussian filtering of the low frequency color plane image received from a preceding level of the plurality of noise cleaning levels, dividing the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels into the low and high frequency color plane images, the low frequency color plane images being generated by downsizing the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels, and the high frequency color plane image being generated by upsizing the low frequency color plane image, and forwarding the low frequency color plane image to the subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level. ### Claim 5 (depends on 4) The method of claim 4, further comprising: in the highest noise cleaning level, bilaterally filtering the low frequency color plane image received from an immediately preceding noise cleaning level, upsizing the low frequency color plane image received from the immediately preceding noise cleaning level; and reconstructing the at least one color plane image by adding high frequency color plane images received from each of the preceding noise cleaning levels to the bilaterally filtered and upsized low frequency color plane image received from the immediately preceding noise cleaning level. ### Claim 6 (depends on 3) The method of claim 3, wherein bilaterally filtering comprises: defining a kernel of a target pixel, the kernel having a square width size and including the target pixel and kernel pixels; subsequently processing each pixel of the at least one color plane image as the target pixel by determining a weighted averaged pixel value for the target pixel, the weighted averaged pixel value for the target pixel being determined by applying weights to values of each kernel pixel and by subsequently averaging the weighted kernel pixel values. ### Claim 7 (depends on 6) The method of claim 6, wherein: a weight for a kernel pixel is determined based on a distance of the kernel pixel from the target pixel and based on a z-score defining a similarity of the value of the kernel pixel to the value of the value of the target pixel; and the distance of the kernel pixel from the target pixel is determined to be consistent with spatial gaps in the color mosaic having the CFA pattern. ### Claim 8 (depends on 7) The method of claim 7, wherein: the weight for the kernel pixel is determined based on: Weights  ( ρ ) = ∑ ρ   ϵ   Kernel  e - ( ( Δ   r 2 · ThrottleR_ML ) 2 + ( Δ   v σ  [ I  ( x , y ) ] · ThrottleS_ML · 2 - level ) 2 ) , wherein ThrottleR_ML defines a Gaussian radius, ThrottleS_ML defines a pixel similarity, Δr defines a distance in pixels from the kernel, I(x,y) defines the target pixel centered in the kernel, Δν defines a code value difference, σ[I(x,y)] defines a noise table, and level defines the noise cleaning level. ### Claim 9 An image processing apparatus comprising: one or more processors in communication with an image sensor; one or more memory devices in communication with the one or more processors; wherein the one or more processors are configured to: receive from the image sensor an image frame including a color mosaic having a color filter array (CFA) pattern; decompose the image frame into at least one color plane image for each of a plurality of color planes; sequentially and separately reduce noise in the at least one color plane image by performing multilevel bilateral noise filtering of the at least one color plane image; and reconstruct the at least one color plane image to generate a noise filtered image frame. ### Claim 10 (depends on 9) The image processing apparatus of claim 9, wherein the plurality of color planes includes at least one of a green color plane, a red color plane, and a blue color plane. ### Claim 11 (depends on 9) The image processing apparatus of claim 9, wherein, to perform the multilevel bilateral noise filtering, the one or more processors are configured to: define a plurality of noise cleaning levels; bilaterally filter the at least one color plane image in each of the plurality of noise cleaning levels; and perform frequency splitting of the at least one color plane image in each of the plurality of noise cleaning levels except for a highest noise cleaning level. ### Claim 12 (depends on 11) The image processing apparatus of claim 11, wherein, to perform the frequency splitting, the one or more processors are configured to: in a first noise cleaning level, perform Gaussian filtering of the at least one color plane image, divide the at least one color plane image into low and high frequency color plane images, the low frequency color plane image being generated by downsizing the Gaussian filtered at least one color plane image, and the high frequency color plane image being generated by upsizing the low frequency color plane image and subtracting respective pixel values of the upsized low frequency color plane from corresponding pixel values of the at least one color plane image, and forward the low frequency color plane image to a subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level; and in at least one noise cleaning level subsequent to the first noise cleaning level and preceding the highest noise cleaning level, perform Gaussian filtering of the low frequency color plane image received from a preceding level of the plurality of noise cleaning levels, divide the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels into the low and high frequency color plane images, the low frequency color plane images being generated by downsizing the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels, and the high frequency color plane image being generated by upsizing the low frequency color plane image, and forward the low frequency color plane image to the subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level. ### Claim 13 (depends on 12) The image processing apparatus of claim 12, wherein the one or more processors are further configured to: in the highest noise cleaning level, bilaterally filter the low frequency color plane image received from an immediately preceding noise cleaning level, upsize the low frequency color plane image received from the immediately preceding noise cleaning level; and reconstruct the at least one color plane image by adding high frequency color plane images received from each of the preceding noise cleaning levels to the bilaterally filtered and upsized low frequency color plane image received from the immediately preceding noise cleaning level. ### Claim 14 (depends on 11) The image processing apparatus of claim 11, wherein, to bilaterally filter the low frequency color plane image, the one or more processors are further configured to: define a kernel of a target pixel, the kernel having a square width size and including the target pixel and kernel pixels; subsequently process each pixel of the at least one color plane image as the target pixel by determining a weighted averaged pixel value for the target pixel, the weighted averaged pixel value for the target pixel being determined by applying weights to values of each kernel pixel and by subsequently averaging the weighted kernel pixel values. ### Claim 15 A non-transitory computer readable storage medium encoded with software comprising computer executable instructions that when executed by one or more processors cause the one or more processors to: receive from the image sensor an image frame including a color mosaic having a color filter array (CFA) pattern; decompose the image frame into at least one color plane image for each of a plurality of color planes; sequentially and separately reduce noise in the at least one color plane image by performing multilevel bilateral noise filtering of the at least one color plane image; and reconstruct the at least one color plane image to generate a noise filtered image frame. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein the plurality of color planes includes at least one of a green color plane, a red color plane, and a blue color plane. ### Claim 17 (depends on 15) The non-transitory computer readable storage medium of claim 15, wherein, to perform the multilevel bilateral noise filtering, the executable instructions further cause the one or more processors to: define a plurality of noise cleaning levels; bilaterally filter the at least one color plane image in each of the plurality of noise cleaning levels; and perform frequency splitting of the at least one color plane image in each of the plurality of noise cleaning levels except for a highest noise cleaning level. ### Claim 18 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein, to perform the frequency splitting, the executable instructions further cause the one or more processors to: in a first noise cleaning level, perform Gaussian filtering of the at least one color plane image, divide the at least one color plane image into low and high frequency color plane images, the low frequency color plane image being generated by downsizing the Gaussian filtered at least one color plane image, and the high frequency color plane image being generated by upsizing the low frequency color plane image and subtracting respective pixel values of the upsized low frequency color plane from corresponding pixel values of the at least one color plane image, and forward the low frequency color plane image to a subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level; and in at least one noise cleaning level subsequent to the first noise cleaning level and preceding the highest noise cleaning level, perform Gaussian filtering of the low frequency color plane image received from a preceding level of the plurality of noise cleaning levels, divide the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels into the low and high frequency color plane images, the low frequency color plane images being generated by downsizing the Gaussian filtered low frequency color plane image from the preceding level of the plurality of noise cleaning levels, and the high frequency color plane image being generated by upsizing the low frequency color plane image, and forward the low frequency color plane image to the subsequent level of the plurality of noise cleaning levels and the high frequency color plane image to the highest noise cleaning level. ### Claim 19 (depends on 18) The non-transitory computer readable storage medium of claim 18, the executable instructions further cause the one or more processors to: in the highest noise cleaning level, bilaterally filter the low frequency color plane image received from an immediately preceding noise cleaning level, upsize the low frequency color plane image received from the immediately preceding noise cleaning level; and reconstruct the at least one color plane image by adding high frequency color plane images received from each of the preceding noise cleaning levels to the bilaterally filtered and upsized low frequency color plane image received from the immediately preceding noise cleaning level. ### Claim 20 (depends on 17) The non-transitory computer readable storage medium of claim 17, wherein, to bilaterally filter the low frequency color plane image, the executable instructions further cause the one or more processors to: define a kernel of a target pixel, the kernel having a square width size and including the target pixel and kernel pixels; subsequently process each pixel of the at least one color plane image as the target pixel by determining a weighted averaged pixel value for the target pixel, the weighted averaged pixel value for the target pixel being determined by applying weights to values of each kernel pixel and by subsequently averaging the weighted kernel pixel values, determine a weight for a kernel pixel based on a distance of the kernel pixel from the target pixel and based on a z-score defining a similarity of the value of the kernel pixel to the value of the value of the target pixel; and determine the distance of the kernel pixel from the target pixel to be consistent with spatial gaps in the color mosaic having the color filter array (CFA) pattern.