# US 2021/0342974 A1 · Texture hallucination for large-scale image super-resolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342974 A1
- Document ID: US20210342974A1
- Status: Published application
- Kind: A1
- Application number: 16/861,688
- Filed: 2020-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Yulun Zhang; Zhifei Zhang; Jose Ignacio Echevarria Vallespi; Zhaowen Wang; Stephen Diverdi
- Web page: https://patentgrep.com/patent/US20210342974A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 3/4046: using neural networks
- G06T 3/4084: in the transform domain, e.g. fast Fourier transform [FFT] domain scaling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for texture hallucination with a large upscaling factor are described. Embodiments of the systems and methods may receive an input image and a reference image, extract an upscaled feature map from the input image, match the input image to a portion of the reference image, wherein a resolution of the reference image is higher than a resolution of the input image, concatenate the upscaled feature map with a reference feature map corresponding to the portion of the reference image to produce a concatenated feature map, and generate a reconstructed image based on the concatenated feature map using a machine learning model trained with a texture loss and a degradation loss, wherein the texture loss is based on a high frequency band filter, and the degradation loss is based on a downscaled version of the reconstructed image.

## Claims

### Claim 1

A method for image upscaling, comprising:
receiving an input image and a reference image;
generating a combined feature map using a machine learning model, the combined feature map including large scale features from the input image and texture features from the reference image; and
generating a reconstructed image based on the combined feature map using the machine learning model, wherein the reconstructed image comprises an upscaled version of the input image with texture detail derived from the reference image and wherein the machine learning model is trained using a texture loss based on the texture detail.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
extracting an upscaled feature map from the input image;
matching the input image to a portion of the reference image, wherein a resolution of the reference image is higher than a resolution of the input image;
concatenating the upscaled feature map with a reference feature map corresponding to the portion of the reference image to produce a concatenated feature map, wherein the combined feature map is generated based on the concatenated feature map.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
extracting a first feature map from the input image;
upscaling the first feature map to produce a first upscaled feature map;
extracting a second feature map from the first upscaled feature map; and
upscaling the second feature map to produce a second upscaled feature map, wherein the upscaled feature map is based on the second upscaled feature map.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
generating a first feature map from the input image using a pre-trained neural network;
generating a second feature map from the reference image using the pre-trained neural network; and
comparing the first feature map to a plurality of portions of the second feature map, wherein the input image is matched to the portion of the reference image based on the comparison.

### Claim 5 (depends on 4)

The method of claim 4, wherein:
the pre-trained neural network comprises a deep convolutional neural network (CNN).

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
extracting the combined feature map from a concatenated feature map.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining that the reference image comprises a texture suitable for the reconstructed image; and
selecting the reference image for generating the reconstructed image based on the determination.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
applying a high frequency band filter to produce a first filtered wavelet transform of a feature map of the reconstructed image and a second filtered wavelet transform of a feature map of the reference image;
comparing the first filtered wavelet transform and the second filtered wavelet transform, wherein the texture loss is based on the comparison.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the machine learning model is further trained using a degradation loss based on a comparison of a downscaled version of the reconstructed image to the input image.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
a resolution of the reconstructed image is more than four times higher than the resolution of the input image.

### Claim 11

A method of training an artificial neural network (ANN) for image upscaling, the method comprising:
extract an upscaled feature map from an input image;
match the input image to a portion of a reference image, wherein a resolution of the reference image is higher than a resolution of the input image;
concatenate the upscaled feature map with a reference feature map corresponding to the portion of the reference image to produce a concatenated feature map;
generate a reconstructed image based on the concatenated feature map using the ANN;
compute a texture loss based on a high frequency band filter and a degradation loss based on a downscaled version of the reconstructed image; and
update the ANN based on the texture loss and the degradation loss.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the texture loss is computed during a first training stage and the degradation loss is computed during a second training stage.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the first training stage is further based on a reconstruction loss comparing the reconstructed image and a ground truth image.

### Claim 14 (depends on 12)

The method of claim 12, wherein:
the second training stage is further based on a perceptual loss comparing the reconstructed image and a ground truth image and an adversarial loss based on the reconstructed image.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
performing a wavelet transform on the reference feature map and on a feature map of the reconstructed image;
compare a high-frequency portion of the wavelet transform of the reference feature map with a high frequency portion of the wavelet transform of the feature map of the reconstructed image, wherein the texture loss is based on the comparison.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
downscaling the reconstructed image to produce the downscaled version of the reconstructed image; and
comparing the downscaled version of the reconstructed image to the input image.

### Claim 17

An apparatus for image upscaling, comprising:
an upscaling component configured to extract an upscaled feature map from an input image;
a matching component configured to match the input image to a portion of a reference image, wherein a resolution of the reference image is higher than a resolution of the input image;
a concatenation component configured to concatenate the upscaled feature map with a reference feature map corresponding to the portion of the reference image to produce a concatenated feature map; and
a reconstruction component configured to generate a reconstructed image based on the concatenated feature map, wherein the reconstruction component is trained with a texture loss and a degradation loss, wherein the texture loss is based on a high frequency band filter, and the degradation loss is based on a downscaled version of the reconstructed image.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein:
the upscaling component comprises a plurality of feature extraction layers and a plurality of upscaling layers.

### Claim 19 (depends on 17)

The apparatus of claim 17, wherein:
the matching component comprises a pre-trained convolutional neural network (CNN).

### Claim 20 (depends on 17)

The apparatus of claim 17, further comprising:
a training component configured to compute the texture loss and the degradation loss and to train the reconstruction component.
