# US 2020/0396398 A1 · Systems and Methods of Spatiotemporal Image Noise Reduction for Multispectral Image Data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0396398 A1 - Document ID: US20200396398A1 - Status: Published application - Kind: A1 - Application number: 16/901,490 - Filed: 2020-06-15 - Published: 2020-12-17 - Earliest claimed: 2019-06-13 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Ilya Romanenko; Alex Hayes - Web page: https://patentgrep.com/patent/US20200396398A1 ## CPC - H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/70: Denoising; Smoothing - H04N 5/33: Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) - H04N 25/60: Noise processing, e.g. detecting, correcting, reducing or removing noise - G06F 2218/04: Denoising - G06T 2207/20221: Image fusion; Image merging - H04N 5/213: Circuitry for suppressing or minimising impulsive noise (for suppressing or minimising disturbance in image signal generation H04N23/81) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Devices, methods, and non-transitory program storage devices for spatiotemporal image noise reduction are disclosed, comprising: maintaining an accumulated image in memory; and obtaining a first plurality of multispectral images (e.g., RGB-IR images). For each image in the first plurality of multispectral images, the method may: calculate a multispectral guide image for the current image; calculate blending weights for the current image; apply the calculated blending weights to each channel of the current image to generate a denoised current image; and update the accumulated image based on pixel differences between the denoised current image and the accumulated image. In some embodiments, additional images (e.g., the accumulated image and/or other images captured prior to or after a given current image) may also be included in the denoising operations for a given current image. Finally, the method may generate a denoised output image for each input image, based on the updated accumulated image. ## Claims ### Claim 1 A device, comprising: a memory; one or more image capture devices; a user interface; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to: maintain an accumulated image in the memory; obtain a first plurality of multispectral images; and for each image in the first plurality of multispectral images, perform the following operations: calculate a multispectral guide image for a current image of the first plurality of multispectral images; calculate a multispectral guide image for the accumulated image; determine first differences between pixels within first blocks of pixels in the multispectral guide image for the current image and second blocks of pixels in the multispectral guide image for the current image; determine second differences between pixels within blocks of pixels in the multispectral guide image for the accumulated image and blocks of pixels in the multispectral guide image for the current image; calculate blending weights for the current image and the accumulated image based, at least in part, on the determined first differences and second differences; denoise the current image based on the calculated blending weights; update the accumulated image based on the denoised current image; and generate a noise-reduced output image based on the updated accumulated image. ### Claim 2 (depends on 1) The device of claim 1, wherein the first plurality of multispectral images comprises RGB-IR images. ### Claim 3 (depends on 2) The device of claim 2, wherein the instructions to determine the first differences and the second differences are executed prior to any decorrelation of RGB image data from IR image data. ### Claim 4 (depends on 1) The device of claim 1, wherein the blocks of pixels comprise neighborhoods of pixels around a central pixel, and wherein the blocks of pixels have dimensions that are determined based, at least in part, on an amount of available pixel lines in memory during execution. ### Claim 5 (depends on 1) The device of claim 1, wherein the multispectral guide image for the accumulated image is formed by: performing a maximum operation across the image channels of the accumulated image at each pixel location. ### Claim 6 (depends on 1) The device of claim 1, wherein at least one of the first differences or the second differences comprise: noise-normalized differences based on an expected noise level. ### Claim 7 (depends on 1) The device of claim 1, wherein the instructions further comprise instructions causing the one or more processors, for a given current image of the first plurality of multispectral images, to: calculate a multispectral guide image for a third image; determine third differences between pixels within blocks of pixels in the multispectral guide image for the third image and blocks of pixels in the multispectral guide image for the given current image; and calculate blending weights for the third image based, at least in part, on the determined third differences. ### Claim 8 (depends on 7) The device of claim 7, wherein the third image comprises an image captured after the given current image. ### Claim 9 (depends on 1) The device of claim 1, wherein the blending weights for the current image and the accumulated multispectral guide image are normalized prior to execution of the instructions to denoise the current image based on the calculated blending weights. ### Claim 10 A non-transitory computer readable medium comprising computer readable instructions configured to cause one or more processors to: maintain an accumulated image in a memory; obtain a first plurality of N-channel multispectral images; and for each image in the first plurality of N-channel multispectral images, perform the following operations: calculate a multispectral guide image for a current image of the first plurality of N-channel multispectral images; calculate blending weights for the current image based, at least in part, on the multispectral guide image for the current image; apply the calculated blending weights to the current image to generate a denoised current image; update the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image; and generate a noise-reduced output image based on the updated accumulated image. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the first plurality of multispectral images comprises RGB-IR images. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, wherein the instructions to apply the calculated blending weights to the current image to generate a denoised current image further comprise instructions to apply the calculated blending weights to each of the N-channels of the current image in parallel. ### Claim 13 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the multispectral guide image for a given current image is formed by: performing a maximum operation across the N image channels of the given current image. ### Claim 14 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the instructions to calculate blending weights for the current image based, at least in part, on the multispectral guide image further comprise instructions to: determine differences between pixels within first blocks of pixels in the multispectral guide image for the current image and second blocks of pixels in the multispectral guide image for the current image. ### Claim 15 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the instructions to update the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image further comprise instructions to: update the accumulated image using a weighting parameter determined based on a noise-normalized difference between the denoised current image and the accumulated image. ### Claim 16 An image processing method, comprising: maintaining an accumulated image in a memory; obtaining a first plurality of N-channel multispectral images; and for each image in the first plurality of N-channel multispectral images, perform the following operations: calculating a multispectral guide image for a current image of the first plurality of N-channel multispectral images; calculating blending weights for the current image based, at least in part, on the multispectral guide image for the current image; applying the calculated blending weights to the current image to generate a denoised current image; updating the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image; and generating a noise-reduced output image based on the updated accumulated image. ### Claim 17 (depends on 16) The method of claim 16, wherein applying the calculated blending weights to the current image to generate a denoised current image further comprises applying the calculated blending weights to each of the N-channels of the current image in parallel. ### Claim 18 (depends on 16) The method of claim 16, wherein the multispectral guide image is formed by: performing a maximum operation across the N image channels of the current image. ### Claim 19 (depends on 16) The method of claim 16, wherein calculating blending weights for the current image based, at least in part, on the multispectral guide image further comprises: determining differences between pixels within first blocks of pixels in the multispectral guide image for the current image and second blocks of pixels in the multispectral guide image for the current image. ### Claim 20 (depends on 16) The method of claim 16, wherein updating the accumulated image based, at least in part, on pixel differences between the denoised current image and the accumulated image further comprises: updating the accumulated image using a weighting parameter determined based on a noise-normalized difference between the denoised current image and the accumulated image.