# US 2020/0401870 A1 · System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401870 A1
- Document ID: US20200401870A1
- Status: Published application
- Kind: A1
- Application number: 17/007,739
- Filed: 2020-08-31
- Published: 2020-12-24
- Earliest claimed: 2017-03-15
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Haoyu REN; Mostafa EL-KHAMY; Jungwon LEE
- Web page: https://patentgrep.com/patent/US20200401870A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Apparatuses and methods of manufacturing same, systems, and methods for generating a convolutional neural network (CNN) are described. In one aspect, a minimal CNN having, e.g., three or more layers is trained. Cascade training may be performed on the trained CNN to insert one or more intermediate layers until a training error is less than a threshold. When cascade training is complete, cascade network trimming of the CNN output from the cascade training may be performed to improve computational efficiency. To further reduce network parameters, convolutional filters may be replaced with dilated convolutional filters with the same receptive field, followed by additional training/fine-tuning.

## Claims

### Claim 1

An apparatus for generating a convolutional neural network (CNN), the apparatus comprising:
one or more non-transitory computer-readable media; and
at least one processor which, when executing instructions stored on the one or more non-transitory computer-readable media, performs the steps of:
starting cascade training on the CNN; and
inserting one or more layers into the CNN where a training error converges or remains higher than a threshold.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the at least one processor, when executing instructions stored on the one or more non-transitory computer-readable media, performs the steps of:
if the training error does not converge, determining whether the training error is less than the threshold;
if the training error is less than the threshold, outputting the cascade trained CNN; and
if the training error is not less than the threshold, starting a new stage.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein a predetermined setting of each new layer in the new stage is randomly initialized with a Gaussian distribution with zero mean and standard deviation σ.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the at least one processor, when executing instructions stored on the one or more non-transitory computer-readable media, performs the steps of:
performing cascade training; and
performing cascade network trimming after performing cascade training.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein cascade network trimming comprises an iterative process of one or more stages, in which each stage comprises:
trimming a set number of layers of the CNN by reducing dimensions of filters at one or more intermediate layers;
if the training error is converging, determining whether all of the layers of the CNN have been trimmed;
if all of the layers of the CNN have been trimmed, outputting the network trimmed CNN; and
if all of the layers of the CNN have not been trimmed, starting a new stage.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein cascade network trimming further comprises:
if the training error is not converging, outputting the CNN at a cascade trimming stage where the training error was last converging.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the cascade training is performed using dilated convolutional filters.

### Claim 9

An apparatus for generating a convolutional neural network (CNN), the apparatus comprising:
one or more non-transitory computer-readable media; and
at least one processor which, when executing instructions stored on the one or more non-transitory computer-readable media, performs the steps of:
performing cascade network trimming of the CNN, the trimming comprising:
trimming a number of layers of the CNN; and
determining whether one or more layers of the CNN have been trimmed.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the cascade network trimming further comprises:
determining whether a training error is converging; and
if the training error is not converging, outputting the CNN at a cascade trimming stage where the training error was last converging.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein each stage of the cascade network trimming further comprises:
fine tuning before determining whether the training error is converging.

### Claim 12 (depends on 9)

The apparatus of claim 9, wherein trimming the number of layers of the CNN comprises, for each layer:
trimming filters that do not meet a certain criteria.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the certain criteria comprises a measure of relative importance.

### Claim 14 (depends on 9)

The apparatus of claim 9, wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.

### Claim 15 (depends on 9)

The apparatus of claim 9, wherein the at least one processor, when executing instructions stored on the one or more non-transitory computer-readable media, performs the step of:
performing cascade training of the CNN before cascade network trimming.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein cascade training comprises an iterative process of one or more stages, in which each stage comprises:
training the CNN;
determining whether a training error is converging; and
if the training error is converging,
inserting a preset number of intermediate layers in the CNN, wherein weights of each new layer are set to a predetermined setting; and
starting a new stage.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein each stage of the cascade training iterative process further comprises:
if the training error is not converging, determining whether the training error is less than the threshold;
if the training error is less than the threshold, outputting the cascade trained CNN; and
if the training error is not less than the threshold, starting a new stage.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the predetermined setting of each new layer is randomly initialized with a Gaussian distribution with zero mean and standard deviation a.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the cascade training is performed using dilated convolutional filters.

### Claim 20

An apparatus for generating a convolutional neural network (CNN), the apparatus comprising:
one or more non-transitory computer-readable media; and
at least one processor which, when executing instructions stored on the one or more non-transitory computer-readable media, performs the steps of:
performing cascade training on the CNN to produce a CNN output; and
performing network trimming of the CNN output.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein cascade training comprises an iterative process of one or more stages, in which each stage comprises:
training the CNN;
determining whether a training error is converging; and
if the training error is converging,
inserting a preset number of intermediate layers in the CNN, wherein weights of each new layer being set to a predetermined setting; and
starting a new stage.

### Claim 22 (depends on 20)

The apparatus of claim 20, wherein the cascade training is performed using dilated convolutional filters.

### Claim 23 (depends on 20)

The apparatus of claim 20, wherein the network trimming comprises an iterative process of one or more stages, in which each stage comprises:
trimming a set number of layers of the CNN by reducing dimensions of filters at one or more intermediate layers;
determining whether a training error is converging; and
if the training error is converging, determining whether all of the layers of the CNN have been trimmed;
if all of the layers of the CNN have been trimmed, outputting the network trimmed CNN; and
if all of the layers of the CNN have not been trimmed, starting a new stage.

### Claim 24 (depends on 20)

The apparatus of claim 20, wherein the CNN is a super-resolution CNN (SRCNN) for processing at least one of images or video.

### Claim 25

A method of manufacturing a chipset, the method comprising:
at least one processor which, when executing instructions stored on one or more non-transitory computer-readable media, performs the steps of:
performing cascade training on a CNN to produce a CNN output; and
performing network trimming of the CNN output.

### Claim 26

A method of testing an apparatus, the method comprising:
testing whether the apparatus has at least one processor which, when executing instructions stored on one or more non-transitory computer-readable media, performs the steps of:
performing cascade training on a CNN to produce a CNN output; and
performing network trimming of the CNN output.