# US 2021/0241429 A1 · Deep residual network for color filter array image denoising

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241429 A1
- Document ID: US20210241429A1
- Status: Published application
- Kind: A1
- Application number: 16/981,866
- Filed: 2020-01-23
- Published: 2021-08-05
- Earliest claimed: 2020-01-23
- Assignees: Baidu USA LLC; Baidu.Com Times Technology (Beijing) Co., Ltd.
- Applicants: Baidu USA, LLC; Baidu.com Times Technology (Beijing) Co., Ltd.
- Inventors: Zhihong PAN; Baopu LI; Hsuchun CHENG; Yingze BAO
- Web page: https://patentgrep.com/patent/US20210241429A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 5/20: using local operators
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- H04N 25/135: {based on four or more different wavelength filter elements}
- G06N 3/08: Learning methods
- G06T 2207/10024: Color image
- G06T 2207/10036: Multispectral image; Hyperspectral image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20224: Image subtraction
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Described herein are embodiments of a deep residual network dedicated to color filter array mosaic patterns. A mosaic stride convolution layer is introduced to match the mosaic pattern of a multispectral filter arrays (MSFA) or a color filter array raw image. Embodiments of a data augmentation using MSFA shifting and dynamic noise are applied to make the model robust to different noise levels. Embodiments of network optimization criteria may be created by using the noise standard deviation to normalize the L1 loss function. Comprehensive experiments demonstrate that embodiments of the disclosed deep residual network outperform the state-of-the-art denoising algorithms in MSFA field.

## Claims

### Claim 1

A computer-implemented method for denoising a noisy image using a deep residual network, the method comprising:
generating, using a first block comprising at least a mosaic stride convolutional layer, a convoluted output image from the noisy image, the mosaic stride convolutional layer comprises multiple strided convolutional channels to sample the noisy image into multiple sampled images with each sampled image filtered by a convolution at a stride into a filtered image, the filtered images from the multiple strided convolutional channels are merged for the generation of the convoluted output image;
generating, using one or more residual blocks in the deep residual network, an intermediate output image from the rectified output, each residual block comprises one or more layers of convolutions and rectified linear units (ReLUs) with a shortcut connection spanning the one or more layers of convolutions and ReLUs;
transforming, using a convolutional layer in the deep residual network, the intermediate output image to an estimated noise image; and
subtracting the estimated noise image from the noisy image to obtain a denoised image.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 wherein the deep residual network is pre-trained using a loss function that normalizes a L1 loss between a plurality of ground-truth noise-free raw images and corresponding denoised images by noise standard deviation.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein the deep residual network is pre-trained using a training data dynamically augmented by adding random noise at different training iterations.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein the deep residual network is pre-trained with shifting a mosaic sampling window for training image by one pixel vertically or horizontally to increase mosaic sampling space.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein the noisy image has a mosaic pattern of w×h with C=w×h spectral channels, the mosaic stride convolutional layer has C=w×h strided convolutional channels, wherein w and h are integer numbers.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5 wherein each strided convolutional channel has a stride with a dimension of w×h and a convolution centered around one of the C spectral channels.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 wherein the multiple sampled images have the same dimension as the noisy image.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1 wherein the first block further comprises one or more parameterized rectified linear unit (PReLU) filters following the mosaic stride convolution layer, the filtered images from the multiple strided convolutional channels are merged and then processed by the one or more PReLU filters such that the convoluted output from the first block is a rectified convoluted output.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1 wherein the first block comprises multiple mosaic stride convolutional layers coupled in series for multiple implementations of sampling, strided convolution, and merging, each mosaic stride convolutional layer is followed by one or more parameterized rectified linear unit (PReLU) filters.

### Claim 10

A method for training a deep residual network for image denoising, the method comprising:
inputting a set of training image pairs, in which a training image pair comprising a noise-free image and a noisy image corresponding to the noise-free image and both the noise-free image and the noisy image have a mosaic pattern, into a deep residual network, the deep residual network comprising a first block, one or more residual blocks, and a final block, the first block comprises a mosaic stride convolutional layer;
generating, using the deep residual network, a denoised image for each noisy image, with steps comprising:
generating, using the mosaic stride convolutional layer, a convoluted output image from a noisy image, the mosaic stride convolutional layer comprises multiple strided convolutional channels to sample the noisy image into multiple sampled images with each sampled image filtered by a convolution at a stride into a filtered image, the filtered images from the multiple strided convolutional channels are merged into the convoluted output image;
generating, using one or more residual blocks in the deep residual network, an intermediate output image from the convoluted output image;
transforming, using a convolutional layer in the deep residual network, the intermediate output image to an estimated noise image; and
subtracting the estimated noise image from the noisy image to obtain the denoised image; and
training the deep residual network using a loss function involving a loss between a plurality of ground-truth noise-free raw images and corresponding denoised images.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10 wherein the loss between a plurality of ground-truth noise-free raw images and corresponding denoised images is a L1 loss, the loss function normalizes the L1 loss by noise standard deviation.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10 wherein noisy images of the set of training image pairs are dynamically augmented from corresponding noise-free images by adding noise to the corresponding noise-free images at different training iterations.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10 wherein noisy images of the set of training image pairs are obtained by shifting a mosaic sampling window for mosaic sampling by one pixel vertically or horizontally to increase mosaic sampling space.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10 wherein the noisy image has a mosaic pattern of w×h, the mosaic stride convolutional layer has C=w×h strided convolutional channels, wherein w and h are integer numbers, each strided convolutional channel has a stride with a dimension of w×h and a convolution.

### Claim 15

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes the steps for denoising a noisy image to be performed comprising:
sampling the noisy image, in a mosaic stride convolutional layer comprising multiple strided convolutional filters, into multiple sampled images;
filtering each of the multiple sampled images with a corresponding strided convolution filter with a convolution kernel at a stride into a filtered image;
merging all filtered images into a convoluted output image;
generating, using one or more residual blocks in the deep residual network, an intermediate output image from the convoluted output image, each residual block comprises one or more layers of convolutions and rectified linear units (ReLUs) with a shortcut connection spanning the one or more layers of convolutions and PReLUs;
transforming, using a convolutional layer, the intermediate output image to an estimated noise image; and
subtracting the estimated noise image from the noisy image to obtain a denoised image.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the noisy image has a mosaic pattern of w×h, the mosaic stride convolutional layer has C=w×h strided convolutional channels, wherein w and h are integer numbers.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium or media of claim 16 wherein the strided convolution filter has the stride with a dimension of w×h.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the multiple sampled images have the same dimension as the noisy image.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the noisy image is captured using a multispectral filter arrays (MSFA) camera or a color filter array (CFA) camera.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium or media of claim 19 wherein each of the strided convolutional filters is centered around one of spectral channels of the noisy image.