# US 2020/0396397 A1 · Multispectral Image Processing System and Method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0396397 A1 - Document ID: US20200396397A1 - Status: Published application - Kind: A1 - Application number: 16/900,621 - Filed: 2020-06-12 - Published: 2020-12-17 - Earliest claimed: 2019-06-13 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Alex Hayes; Ilya Romanenko - Web page: https://patentgrep.com/patent/US20200396397A1 ## CPC - G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215) - G06T 5/50: using two or more images, e.g. averaging or subtraction - H04N 23/11: for generating image signals from visible and infrared light wavelengths - G06T 2207/10024: Color image - G06T 2207/10048: Infrared image - G06T 2207/20221: Image fusion; Image merging - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Devices, methods, and non-transitory program storage devices are disclosed herein to provide improved multi-spectral image processing techniques for generating an enhanced output image, the techniques comprising: obtaining an N-channel (e.g., multispectral) input image; determining fusion weights and fallback weights (e.g., relative intensity weights) for each of the N-channels of the input image; blending the fusion and fallback weights based on an amount of gradient information to generate blended weights; modulating the blended weights for a plurality of frequency band representations of the input image; applying the modulated blended weights to the corresponding frequency band representations of the input image to generate a plurality of output image frequency band representations; producing an output luma image, based on the plurality of output image frequency band representations; and generating an output RGB image, based on the output luma image, which may then, e.g., be displayed to a user or stored to non-volatile memory. ## Claims ### Claim 1 A device, comprising: a memory; one or more image capture devices; a user interface; and one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to: obtain an N-channel input image; determine fusion weights and fallback weights for each of the N-channels of the input image; blend the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights; modulate the blended weights for a plurality of frequency band representations of the input image; apply the modulated blended weights to the corresponding frequency bend representations of the input image to generate a plurality of output image frequency band representations; and produce an output luma image, based on the plurality of output image frequency band representations. ### Claim 2 (depends on 1) The device of claim 1, wherein the N-channel input image comprises an RGB-IR image. ### Claim 3 (depends on 1) The device of claim 1, wherein the instructions further comprise instructions to: generate an output RGB image based on the output luma image. ### Claim 4 (depends on 1) The device of claim 1, wherein the fusion weights are further determined based, at least in part, on one or more of the following: a principal characteristic vector of an outer product of Jacobian matrices of the input image's N-channels; a local entropy estimate from an input image channel; or a gradient magnitude estimate from an input image channel. ### Claim 5 (depends on 1) The device of claim 1, wherein at least one of the plurality of frequency band representations comprises a high frequency band representation, and wherein the fusion weights are further determined based, at least in part, on information in the high frequency band representation. ### Claim 6 (depends on 1) The device of claim 1, wherein the frequency band representations of the input image are created by a multiscale decomposition process. ### Claim 7 (depends on 1) The device of claim 1, wherein the amount of gradient information at a pixel in the input image is determined based, at least in part, on: a size of a largest eigenvalue of a Jacobian matrix of gradients for the input image; and a noise estimate for the pixel. ### Claim 8 (depends on 1) The device of claim 1, wherein the fallback weight for a given input image channel comprises: a weight based on the input intensity of the given input image channel relative to a summation of the input intensities of the N-channels of the input image. ### Claim 9 A non-transitory computer readable medium comprising computer readable instructions configured to cause one or more processors to: obtain an N-channel input image; determine fusion weights and fallback weights for each of the N-channels of the input image; blend the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights; modulate the blended weights for a plurality of frequency band representations of the input image; apply the modulated blended weights to the corresponding frequency bend representations of the input image to generate a plurality of output image frequency band representations; and produce an output luma image, based on the plurality of output image frequency band representations. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the N-channel input image comprises an RGB-IR image. ### Claim 11 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the plurality of frequency band representations comprises a high frequency band representation, and wherein the fusion weights are further determined based, at least in part, on information in the high frequency band representation. ### Claim 12 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the frequency band representations of the input image are created by a multiscale decomposition process. ### Claim 13 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the amount of gradient information at a pixel in the input image is determined based, at least in part, on: a size of a largest eigenvalue of a Jacobian matrix of gradients for the input image; and a noise estimate for the pixel. ### Claim 14 (depends on 9) The non-transitory computer readable medium of claim 9, wherein the fallback weight for a given input image channel comprises: a weight based on the input intensity of the given input image channel relative to a summation of the input intensities of the N-channels of the input image. ### Claim 15 An image processing method, comprising: obtaining an N-channel input image; determining fusion weights and fallback weights for each of the N-channels of the input image; blending the fusion and fallback weights based on an amount of gradient information in the input image to generate blended weights; modulating the blended weights for a plurality of frequency band representations of the input image; applying the modulated blended weights to the corresponding frequency bend representations of the input image to generate a plurality of output image frequency band representations; and producing an output luma image, based on the plurality of output image frequency band representations. ### Claim 16 (depends on 15) The method of claim 15, wherein the N-channel input image comprises an RGB-IR image. ### Claim 17 (depends on 15) The method of claim 15, further comprising: generating an output RGB image based on the output luma image. ### Claim 18 (depends on 17) The method of claim 17, wherein generating an output RGB image based on the output luma image further comprises: determining original color differences for pixels in the input image; modulating the determined original color differences for pixels in the input image; and adding the modulated determined original color differences for the pixels in the input image to the corresponding pixels in the output luma image to generate the output RGB image. ### Claim 19 (depends on 15) The method of claim 15, wherein the frequency band representations of the input image are created by a difference of Gaussians (DoG) pyramid operation. ### Claim 20 (depends on 15) The method of claim 15, wherein the method is performed, at least in part, by a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC).