# US 2021/0241421 A1 · Systems and methods for multispectral image demosaicking using deep panchromatic image guided residual interpolation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241421 A1
- Document ID: US20210241421A1
- Status: Published application
- Kind: A1
- Application number: 16/960,304
- Filed: 2019-07-05
- Published: 2021-08-05
- Earliest claimed: 2019-07-05
- Assignees: Baidu USA LLC; Baidu.Com Times Technology (Beijing) Co., Ltd.
- Applicants: Baidu USA, LLC; Baidu.com Times Technology (Beijing) Co., Ltd.
- Inventors: Zhihong PAN; Baopu LI; Yingze BAO; Hsuchun CHENG
- Web page: https://patentgrep.com/patent/US20210241421A1

## CPC

- G06T 3/4015: Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06T 5/73: Deblurring; Sharpening
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Described herein are systems and embodiments for multispectral image demosaicking using deep panchromatic image guided residual interpolation. Embodiments of a ResNet-based deep learning model are disclosed to reconstruct the full-resolution panchromatic image from multispectral filter array (MSFA) mosaic image. In one or more embodiments, the reconstructed deep panchromatic image (DPI) is deployed as the guide to recover the full-resolution multispectral image using a two-pass guided residual interpolation methodology. Experiment results demonstrate that the disclosed method embodiments outperform some state-of-the-art conventional and deep learning demosaicking methods both qualitatively and quantitatively.

## Claims

### Claim 1

A computer-implemented method for demosaicking a multispectral image from a multispectral filter arrays (MSFA) sensor with multiple sub-bands, the method comprising:
using a deep neural network and the multispectral image to reconstruct a full resolution deep panchromatic image (DPI) of the multispectral image;
for each sub-band of the multispectral image, performing at least one guided residual interpolation based on the reconstructed DPI to obtain a demosaicked sub-band image; and
forming a demosaicked multispectral image comprising the demosaicked sub-band images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the step of for each sub-band of the multispectral image, performing at least one-pass guided residual interpolation based on the reconstructed DPI to obtain the demosaicked sub-band image comprise:
subsampling the DPI relative to the sub-band to obtain a subsampled DPI;
obtaining a subsampled residual image from a difference between a sub-band image corresponding to the sub-band and the subsampled DPI;
using the DPI as a guide image in a first guided interpolation on the subsampled residual image to obtain a demosaicked residual image; and
adding the demosaicked residual image to the DPI to obtain a first-pass demosaicked sub-band image.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2 wherein the step of for each sub-band of the MSFA image, performing at least one guided residual interpolation based on the reconstructed DPI to obtain the demosaicked sub-band image further comprising the steps of:
subsampling the first-pass demosaicked sub-band image relative to the sub-band to obtain a second-pass subsampled image;
obtaining a second-pass subsampled residual image from a difference between sub-band image and the second-pass subsampled image;
using the first-pass demosaicked sub-band image as a guide image in a second guided interpolation on the second-pass subsampled residual image to obtain a second-pass demosaicked residual image; and
adding the second-pass residual image to the first-pass demosaicked sub-band image to obtain a second-pass demosaicked sub-band image.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein the deep neural network is a convolutional neural network (CNN), the CNN comprises:
a projection layer transforming the multispectral image to multiple channels;
one or more middle blocks to generate an intermediate output based on the multiple channels, each block has two layers with each layer comprising a convolutional and a rectified linear unit (ReLU) filters, and a skip connection between an input and an output of each block;
a final convolutional layer projecting the intermediate output to a 2D residual image; and
a clipping module clipping a residual subtracted image obtained by subtracting the 2D residual image from the multispectral image to a valid pixel value range for the DPI.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4 wherein the CNN is pre-trained using a training data set to minimize a loss between the reconstructed DPI and a reference panchromatic image averaged from all bands of a ground-truth multispectral image.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 wherein during the pre-training, a sharpen filter is applied to both the reconstructed DPI and the reference panchromatic image before calculating the loss.

### Claim 7 (depends on 4)

The computer-implemented method of claim 4 wherein a padding scheme is used in the first block for a limited rolling shift of a first MSFA pattern on each side.

### Claim 8 (depends on 3)

The computer-implemented method of claim 3 wherein the first guided interpolation and the second guided interpolation comprise steps of:
for a MSFA pattern of m×n, m and n being positive integer numbers, using a sliding window size (m+1)×(n+1) with m and n sliding steps in corresponding directions in the MSFA pattern respectively, the MSFA pattern corresponds to the subsampled residual image in the first guided interpolation or the second-pass subsampled residual image in the second guided interpolation;
for each sliding step of the sliding window, using only pixels from a sparse grid in each sub-band to estimate coefficients;
for each pixel in the sparse grid, obtaining linear interpolation coefficients using the average of the estimated coefficients for all sliding steps;
obtaining a full resolution coefficients for the sub-band of the multispectral image from the sparse grid using bilinear interpolation; and
obtaining a full resolution image for the sub-band from a linear transformation using the reconstructed DPI as a guided image and the obtained full resolution coefficients in the first guided interpolation, or from a linear transformation using the first-pass demosaicked sub-band image as a guided image and the obtained full resolution coefficients in the second guided interpolation, the full resolution image is the demosaicked residual image in the first guided interpolation, or the second-pass demosaicked residual image in the second guided interpolation.

### Claim 9

A computer-implemented method for demosaicking a sub-band image in a multispectral image captured by a multispectral filter arrays (MSFA) sensor using one or more processors to cause steps to be performed comprising:
using a deep neural network and the multispectral image to reconstruct a full resolution deep panchromatic image (DPI) of the multispectral image;
subsampling the DPI relative to the sub-band to obtain a subsampled DPI;
obtaining a subsampled residual image from a difference between the sub-band image and the subsampled DPI;
using the DPI as a guide image in a guided interpolation on the subsampled residual image to obtain a demosaicked residual image; and
adding the demosaicked residual image to the DPI to obtain a first-pass demosaicked sub-band image.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9 further comprising:
subsampling the first-pass demosaicked sub-band image relative to the sub-band to obtain a second-pass subsampled image;
obtaining a second-pass subsampled residual image from a difference between sub-band image and the second-pass subsampled image;
using the first-pass demosaicked sub-band image as a guide image in a second guided interpolation on the second-pass subsampled residual image to obtain a second-pass demosaicked residual image; and
adding the second-pass residual image to the first-pass demosaicked sub-band image to obtain a second-pass demosaicked sub-band image.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10 wherein the guided interpolation and the second guided interpolation comprise steps of:
for a MSFA pattern of m×n, m and n being positive integer numbers, using a sliding window size (m+1)×(n+1) with m and n sliding steps in corresponding directions in the MSFA pattern respectively, the MSFA pattern corresponds to the subsampled residual image in the guided interpolation or the second-pass subsampled residual image in the second guided interpolation;
for each sliding step of the sliding window, using only pixels from a sparse grid in each sub-band to estimate coefficients;
for each pixel in the sparse grid, obtaining linear interpolation coefficients using the average of the estimated coefficients for all sliding steps;
obtaining a full resolution coefficients for the sub-band of the multispectral image from the sparse grid using bilinear interpolation; and
obtaining a full resolution image for the sub-band from a linear transformation using the reconstructed DPI as a guided image and the obtained full resolution coefficients in the guided interpolation, or from a linear transformation using the first-pass demosaicked sub-band image as a guided image and the obtained full resolution coefficients in the second guided interpolation, the full resolution image is the demosaicked residual image in the guided interpolation, or the second-pass demosaicked residual image in the second guided interpolation.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9 wherein the deep neural network is a convolutional neural network (CNN), the CNN comprises:
a projection layer transforming the multispectral image to multiple channels;
one or more middle blocks to generate an intermediate output based on the multiple channels, each block has two layers with each layer comprising a convolutional and a rectified linear unit (ReLU) filters, and a skip connection between each block;
a final convolutional layer projecting the intermediate output to a 2D residual image; and
a clipping module clipping a residual subtracted image obtained by subtracting the 2D residual image from the multispectral image to a valid pixel value range for the DPI.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12 wherein the CNN is pre-trained using a training data set to minimize a loss between the reconstructed DPI and a reference panchromatic image averaged from all bands of a ground-truth multispectral image.

### Claim 14 (depends on 13)

The computer-implemented method of claim 13 wherein during the pre-training, a sharpen filter is applied to both the reconstructed DPI and the reference panchromatic image before calculating the loss.

### Claim 15

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps for demosaicking a multispectral image, from a multispectral filter arrays (MSFA) sensor, with multiple sub-bands to be performed comprising:
reconstructing, using a deep neural network, the multispectral image to a full resolution deep panchromatic image (DPI);
for each sub-band of the multispectral image, performing at least one guided residual interpolation based on the reconstructed DPI to obtain a demosaicked sub-band image; and
forming a demosaicked multispectral image comprising the demosaicked sub-band images.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein performing at least one-pass guided residual interpolation based on the reconstructed DPI to obtain the demosaicked sub-band image comprise:
subsampling the DPI relative to the sub-band to obtain a subsampled DPI;
obtaining a subsampled residual image from a difference between a sub-band image corresponding to the sub-band and the subsampled DPI;
using the DPI as a guide in a first guided interpolation on the subsampled residual image to obtain a demosaicked residual image; and
adding the demosaicked residual image to the DPI to obtain a first-pass demosaicked sub-band image.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein performing at least one-pass guided residual interpolation based on the reconstructed DPI to obtain the demosaicked sub-band image further comprising:
subsampling the first-pass demosaicked sub-band image relative to the sub-band to obtain a second-pass subsampled image;
obtaining a second-pass subsampled residual image from a difference between sub-band image and the second-pass subsampled image;
using the first-pass demosaicked sub-band image as a guide in a second guided interpolation on the second-pass subsampled residual image to obtain a second-pass demosaicked residual image; and
adding the second-pass residual image to the first-pass demosaicked sub-band image to obtain a second-pass demosaicked sub-band image.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium or media of claim 17 wherein the first guided interpolation or the second guided interpolation comprising steps of:
for a MSFA pattern of m×n, m and n being positive integer numbers, using a sliding window size (m+1)×(n+1) with m and n sliding steps in corresponding directions in the MSFA pattern respectively, the MSFA pattern corresponds to the subsampled residual image in the first guided interpolation or the second-pass subsampled residual image in the second guided interpolation;
for each sliding step of the sliding window, using only pixels from a sparse grid in each sub-band to estimate coefficients;
for each pixel in the sparse grid, obtaining linear interpolation coefficients using the average of the estimated coefficients for all sliding steps;
obtaining a full resolution coefficients for the sub-band of the multispectral image from the sparse grid using bilinear interpolation; and
obtaining a full resolution image for the sub-band from a linear transformation using the reconstructed DPI as a guided image and the obtained full resolution coefficients in the first guided interpolation, or from a linear transformation using the first-pass demosaicked sub-band image as a guided image and the obtained full resolution coefficients in the second guided interpolation, the full resolution image is the demosaicked residual image in the first guided interpolation, or the second-pass demosaicked residual image in the second guided interpolation.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium or media of claim 18 wherein the deep neural network is a convolutional neural network (CNN), the CNN comprises:
a projection layer transforming the multispectral image to multiple channels;
one or more middle blocks to generate an intermediate output based on the multiple channels, each block has two layers with each layer comprising a convolutional and a rectified linear unit (ReLU) filters, and a skip connection between each block;
a final convolutional layer projecting the intermediate output to a 2D residual image; and
a clipping module clipping a residual subtracted image obtained by subtracting the 2D residual image from the multispectral image to a valid pixel value range for the DPI.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium or media of claim 19 wherein the two layers of each block have different kernel size.

### Claim 21 (depends on 1)

A system for demosaicking a multispectral image from a multispectral filter arrays (MSFA) sensor with multiple sub-bands, the system comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform the method of claim 1.

### Claim 22 (depends on 9)

A system for demosaicking a sub-band image in a multispectral image captured by a multispectral filter arrays (MSFA) sensor, the system comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform the method claim 9.
