# US 10,482,576 B2 · Systems and methods for multi-spectral image super-resolution > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,576 B2 - Document ID: US10482576B2 - Status: Granted - Kind: B2 - Application number: 15/924,559 - Filed: 2018-03-19 - Published: 2019-09-19 - Granted: 2019-11-19 - Earliest claimed: 2018-03-19 - Assignees: Mitsubishi Electric Research Laboratories, Inc. - Applicants: Mitsubishi Electric Research Laboratories, Inc. - Inventors: Dehong Liu; Bihan Wen; Ulugbek Kamilov; Hassan Mansour; Petros Boufounos - Web page: https://patentgrep.com/patent/US10482576B2 ## CPC - G06T 3/4061: by injecting details from different spectral ranges - G06T 2207/10036: Multispectral image; Hyperspectral image - G06T 2207/20081: Training; Learning - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods for image processing for increasing resolution of a multi-spectral image. Accept a multi-spectral image including a set of images of a scene. A memory to store a set of dictionaries trained for different channels, and a set of filters trained for the different channels. A hardware processor is to process the set of images of the different channels with the set of filters, and to fuse, for each channel, the set of structures, to produce a set of fused structures. Wherein a fused structure of the channel is fused as a weighted combination of the set of structures using weights corresponding to the channel, such that the fused structures of different channels are combined with different weights. To process the set of fused structures with corresponding dictionaries from the set of dictionaries, to produce a super-resolution multi-spectral image. An output interface to render the super-resolution multi-spectral image. ## Claims ### Claim 1 An imaging system for increasing resolution of a multi-spectral image, comprising: an input interface to accept a multi-spectral image including a set of images of a scene generated by sensors, each image represents a channel defining a frequency band, such that an image of a channel has its frequencies within a frequency band of the channel; a memory to store a set of synthesis dictionaries trained for different channels, such that a synthesis dictionary trained for a channel convolved with a structure of the image, produces a high-resolution image of the channel having a resolution greater than a resolution of the image of the channel; and a set of filters (analysis dictionaries) trained for the different channels, such that a filter (analysis dictionary) trained for a channel convolved with each image in the set of images produces the structure of the image of the channel; a hardware processor is to process the set of images of the different channels with the set of filters (analysis dictionaries) to produce a set of structures; fuse, for each channel, the set of structures, to produce a set of fused structures, wherein a fused structure of the channel is fused as a weighted combination of the set of structures using weights corresponding to the channel, such that the fused structures of different channels are combined with different weights; process the set of fused structures with corresponding dictionaries from the set of dictionaries, to produce a super-resolution multi-spectral image; and an output interface to render the super-resolution multi-spectral image. ### Claim 2 (depends on 1) The imaging system of claim 1, wherein each filter (analysis dictionary) in the set of filters and each dictionary in the set of dictionaries are trained using different images of the different channels. ### Claim 3 (depends on 2) The imaging system of claim 2, wherein the set of synthesis dictionaries and the set of filters (analysis dictionaries) are jointly trained to increase an average peak signal-to-noise ratio over all training images in the different channels. ### Claim 4 (depends on 3) The imaging system of claim 3, wherein the set of dictionaries and the set of filters are jointly trained by updating current values of the set of dictionaries and the set of filters in a current iteration, using previous values of the set of dictionaries and the set of filters learned during previous iterations. ### Claim 5 (depends on 1) The imaging system of claim 1, wherein the memory stores a set of thresholds trained for the channels, such that the hardware processor imposes sparsity on the structure of the image in the channel, by thresholding values of the structure of the image of the channel with a corresponding threshold. ### Claim 6 (depends on 1) The imaging system of claim 1, wherein the set of dictionaries are convolutional dictionaries. ### Claim 7 (depends on 1) The imaging system of claim 1, wherein the set of dictionaries, the set of filters (analysis dictionaries), and a set of weights for fusing structures of each channel are jointly trained using different images of the different channels. ### Claim 8 (depends on 1) The imaging system of claim 1, wherein the set of weights are incorporated into one or combination of the set of filters (analysis dictionaries) and the set of synthesis dictionaries, such that a combination of the convolutions of the set of synthesis dictionaries with the corresponding structures includes the fusion of the structures of the different images with corresponding weights. ### Claim 9 (depends on 1) The imaging system of claim 1, wherein the weights corresponding to the channel are determined by maximizing the average peak signal-to-noise ratio over all training images and channels. ### Claim 10 A method for image processing for increasing resolution of a multi-spectral image, comprising: acquiring a multi-spectral image including a set of images of a scene generated by sensors and either acquired by an input interface or from a computer readable memory, each image represents a channel defining a frequency band, such that an image of a channel has its frequencies within a frequency band of the channel; storing in a database of the computer readable memory, a set of dictionaries trained for different channels, such that a dictionary trained for a channel convolved with a structure of the image, produces a high-resolution image of the channel having a resolution greater than a resolution of the image of the channel; storing in another database of the computer readable memory, a set of filters trained for the different channels, such that a filter trained for a channel convolved with each image in the set of images, produces the structure of the image of the channel; processing the set of images of the different channels with the stored set of filters to produce a set of structures using a hardware processor in communication with the computer readable memory and the input interface; using the hardware processor to fuse, for each channel, the set of structures to produce a set of fused structures, wherein a fused structure of the channel is fused as a weighted combination of the set of structures using weights corresponding to the channel, such that the fused structures of the different channels are combined with different weights, processing the set of fused structures with corresponding dictionaries from the set of dictionaries, to produce a super-resolution multi-spectral image of the scene; and outputting the super-resolution multi-spectral image by an output interface to a communication network, or storing the super-resolution multi-spectral image in the computer readable memory. ### Claim 11 (depends on 10) The method according to claim 10, wherein each filter in the set of filters and each dictionary in the set of dictionaries are trained using different images of the different channels. ### Claim 12 (depends on 11) The method of claim 11, wherein the set of dictionaries and the set of filters are jointly trained to increase an average peak signal-to-noise ratio over all training images in the different channels. ### Claim 13 (depends on 12) The method of claim 12, wherein the set of synthesis dictionaries and the set of filters (analysis dictionaries) are jointly trained by updating current values of the dictionaries and the set of filters in a current iteration using previous values of the dictionaries and the set of filters learned during previous iterations. ### Claim 14 (depends on 12) The method of claim 12, wherein the computer readable memory stores a set of thresholds trained for the channels, such that the hardware processor imposes sparsity on the structure of the image in the channel, by thresholding values of the structure of the image of the channel with a corresponding threshold. ### Claim 15 (depends on 10) The method of claim 10, wherein the set of dictionaries are convolutional dictionaries. ### Claim 16 (depends on 10) The method of claim 10, wherein the set of dictionaries, the set of filters, and a set of weights for fusing structures of each channel are jointly trained using different images of different channels. ### Claim 17 (depends on 10) The method of claim 10, wherein the set of weights are incorporated into one or combination of the set of filters and the set of dictionaries, such that a combination of the convolutions of the set of dictionaries with the corresponding structures includes the fusion of the structures of different images with corresponding weights. ### Claim 18 A non-transitory computer readable storage medium embodied thereon a program executable by a computer for performing a method, the method is for image processing of images of a scene, comprising: acquiring a multi-spectral image including a set of images of a scene generated by sensors and either acquired by an input interface or from the storage medium, each image represents a channel defining a frequency band, such that an image of a channel has its frequencies within a frequency band of the channel; storing in a database of the storage medium, a set of dictionaries trained for different channels, such that a dictionary trained for a channel convolved with a structure of the image, produces a high-resolution image of the channel having a resolution greater than a resolution of the image of the channel; storing in another database of the storage medium, a set of filters trained for the different channels, such that a filter trained for a channel convolved with each image in the set of images, produces the structure of the image of the channel; processing the set of images of the different channels with the stored set of filters to produce a set of structures using the computer in communication with the storage medium and the input interface; using the computer to fuse, for each channel, the set of structures to produce a set of fused structures, wherein a fused structure of the channel is fused as a weighted combination of the set of structures using weights corresponding to the channel, such that the fused structures of the different channels are combined with different weights; processing the set of fused structures with corresponding dictionaries from the set of dictionaries, to produce a super-resolution multi-spectral image of the scene; and outputting the super-resolution multi-spectral image by an output interface to a communication network, or storing the super-resolution multi-spectral image in the storage medium.