# US 11,908,103 B2 · Multi-scale-factor image super resolution with micro-structured masks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,103 B2
- Document ID: US11908103B2
- Status: Granted
- Kind: B2
- Application number: 17/363,280
- Filed: 2021-06-30
- Published: 2022-02-17
- Granted: 2024-02-20
- Earliest claimed: 2020-08-14
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Wei Jiang; Wei Wang; Shan Liu
- Web page: https://patentgrep.com/patent/US11908103B2

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06N 3/063: using electronic means
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

There is included a method and apparatus comprising computer code configured to cause a processor or processors to perform obtaining an input low resolution (LR) image comprising a height, a width, and a number of channels, implementing a feature learning deep neural network (DNN) configured to compute a feature tensor based on the input LR image, generating, by an upscaling DNN, a high resolution (HR) image, having a higher resolution than the input LR image, based on the feature tensor computed by the feature learning DNN, wherein a networking structure of the upscaling DNN differs depending on different scale factors, and wherein a networking structure of the feature learning DNN is a same structure for each of the different scale factors.

## Claims

### Claim 1

A method for image processing, the method performed by at least one processor and comprising:
obtaining an input low resolution (LR) image comprising a height, a width, and a number of channels;
implementing a feature learning deep neural network (DNN) configured to compute a feature tensor based on the input LR image;
generating, by an upscaling DNN, a high resolution (HR) image, having a higher resolution than the input LR image, based on the feature tensor computed by the feature learning DNN,
wherein a networking structure of the upscaling DNN differs depending on different scale factors, and
wherein a networking structure of the feature learning DNN is a same structure for each of the different scale factors.

### Claim 2 (depends on 1)

The method according to claim 1, further, at a test stage, comprising:
generating masked weight coefficients for the feature learning DNN based on a target scale factor; and
selecting a subnetwork of the upscaling DNN for the target scale factor based on selected weight coefficients and computing the feature tensor through inference computation.

### Claim 3 (depends on 2)

The method according to claim 2, further comprising:
generating the HR image based on passing the feature tensor through an upscaling module using the selected weight coefficients.

### Claim 4 (depends on 1)

The method according to claim 1,
wherein weight coefficients of at least one of the feature learning DNN and the upscaling DNN comprise a 5-dimensional (5D) tensor with a size of c1, k1, k2, k3, c2,
wherein an input of a layer of the at least one of the feature learning DNN and the upscaling DNN comprises a 4-dimensional (4D) tensor A with a size of h1, w1, d1, c1,
wherein an output of the layer is a 4D tensor B with a size of h2, w2, d2, c2, and
wherein each of c1, k1, k2, k3, c2, h1, w1, d1, c1, h2, w2, d2, and c2 are integer numbers greater than or equal to 1,
wherein h1, w1, d1 are a height, a weight, and a depth of the tensor A,
wherein h2, w2, d2 are a height, a weight, and a depth of the tensor B,
wherein c1 and c2 are numbers of input and output channels respectively, and
wherein k1, k2, k3 are sizes of a convolution kernel and correspond to height, weight, and depth axes respectively.

### Claim 5 (depends on 4)

The method according to claim 4, further comprising:
reshaping the 5D tensor to a 3-dimensional (3D) tensor; and
reshaping the 5D tensor to a 2-dimensional (2D) matrix.

### Claim 6 (depends on 5)

The method according to claim 5,
wherein the 3D tensor is of a size c′1, c′2, k, where c′1×c′2×k=c1×c2×k1×k2×k3, and
wherein the 2D matrix is of a size c′1, c′2, where c′1×c′2=c1×c2×k1×k2×k3.

### Claim 7 (depends on 6)

The method according to claim 6, further, at a training stage, comprising:
fixing weight coefficients that are masked;
obtaining updated weight coefficients based on a learning process on weights of the feature learning DNN and weights of the upscaling DNN through a weight filling module; and
conducting, based on the updated weight coefficients, a micro-structured pruning process to obtain a model instance and masks.

### Claim 8 (depends on 7)

The method according to claim 7,
wherein the learning process comprises reinitializing the weight coefficients that have zero values by setting those weight coefficients to any of random initial values and corresponding weights of a previously learned model.

### Claim 9 (depends on 7)

The method according to claim 7, wherein the micro-structured pruning process comprises:
computing losses for each of a plurality of micro-structured blocks of at least one of the 3D tensor and the 2D matrix; and
ranking the micro-structured blocks based on the computed losses.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the micro-structured pruning process further comprises:
determining to stop the micro-structured pruning process based on whether a distortion loss reaches a threshold.

### Claim 11

An apparatus for image processing, the apparatus comprising:
at least one memory configured to store computer program code;
at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code including:
obtaining code configured to cause the at least one processor to obtain an input low resolution (LR) image comprising a height, a width, and a number of channels;
implementing code configured to cause the at least one processor to implement a feature learning deep neural network (DNN) configured to compute a feature tensor based on the input LR image;
generating code configured to cause the at least one processor to generate, by an upscaling DNN, a high resolution (HR) image, having a higher resolution than the input LR image, based on the feature tensor computed by the feature learning DNN,
wherein a networking structure of the upscaling DNN differs depending on different scale factors, and
wherein a networking structure of the feature learning DNN is a same structure for each of the different scale factors.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the generating code is further, at a test stage, code configured to cause the at least one processor to:
generate masked weight coefficients for the feature learning DNN based on a target scale factor; and
select a subnetwork of the upscaling DNN for the target scale factor based on selected weight coefficients and computing the feature tensor through inference computation.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein generating the HR image is based on passing the feature tensor through an upscaling module using the selected weight coefficients.

### Claim 14 (depends on 11)

The apparatus according to claim 11,
wherein weight coefficients of at least one of the feature learning DNN and the upscaling DNN comprise a 5-dimensional (5D) tensor with a size of c1, k1, k2, k3, c2,
wherein an input of a layer of the at least one of the feature learning DNN and the upscaling DNN comprises a 4-dimensional (4D) tensor A with a size of h1, w1, d1, c1,
wherein an output of the layer is a 4D tensor B with a size of h2, w2, d2, c2, and
wherein each of c1, k1, k2, k3, c2, h1, w1, d1, c1, h2, w2, d2, and c2 are integer numbers greater than or equal to 1,
wherein h1, w1, d1 are a height, a weight, and a depth of the tensor A,
wherein h2, w2, d2 are a height, a weight, and a depth of the tensor B,
wherein c1 and c2 are numbers of input and output channels respectively, and
wherein k1, k2, k3 are sizes of a convolution kernel and correspond to height, weight, and depth axes respectively.

### Claim 15 (depends on 14)

The apparatus according to claim 14, further comprising:
reshaping code configured to cause the at least one processor to reshape the 5D tensor to a 3-dimensional (3D) tensor and to reshape the 5D tensor to a 2-dimensional (2D) matrix.

### Claim 16 (depends on 15)

The apparatus according to claim 15,
wherein the 3D tensor is of a size c′1, c′2, k, where c′1×c′2×k=c1×c2×k1×k2×k3, and
wherein the 2D matrix is of a size c′1, c′2, where c′1×c′2=c1×c2×k1×k2×k3.

### Claim 17 (depends on 16)

The apparatus according to claim 16, further, at a training stage, comprising:
fixing code configured to cause the at least one processor to fix weight coefficients that are masked;
obtaining code configured to cause the at least one processor to obtain updated weight coefficients based on a learning process on weights of the feature learning DNN and weights of the upscaling DNN through a weight filling module; and
conducting code configured to cause the at least one processor to conduct, based on the updated weight coefficients, a micro-structured pruning process to obtain a model instance and masks.

### Claim 18 (depends on 17)

The apparatus according to claim 17,
wherein the learning process comprises reinitializing the weight coefficients that have zero values by setting those weight coefficients to any of random initial values and corresponding weights of a previously learned model.

### Claim 19 (depends on 17)

The apparatus according to claim 17, wherein the micro-structured pruning process comprises:
computing losses for each of a plurality of micro-structured blocks of at least one of the 3D tensor and the 2D matrix; and
ranking the micro-structured blocks based on the computed losses.

### Claim 20

A non-transitory computer readable medium storing a program causing a computer to execute a process, the process comprising:
obtaining an input low resolution (LR) image comprising a height, a width, and a number of channels;
implementing a feature learning deep neural network (DNN) configured to compute a feature tensor based on the input LR image;
generating, by an upscaling DNN, a high resolution (HR) image, having a higher resolution than the input LR image, based on the feature tensor computed by the feature learning DNN,
wherein a networking structure of the upscaling DNN differs depending on different scale factors, and
wherein a networking structure of the feature learning DNN is a same structure for each of the different scale factors.