# US 10,769,764 B2 · Hierarchical scale matching and patch estimation for image style transfer with arbitrary resolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,764 B2
- Document ID: US10769764B2
- Status: Granted
- Kind: B2
- Application number: 16/271,058
- Filed: 2019-02-08
- Published: 2020-08-13
- Granted: 2020-09-08
- Earliest claimed: 2019-02-08
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Chen Fang; Zhe Lin; Zhaowen Wang; Yulun Zhang; Yilin Wang; Jimei Yang
- Web page: https://patentgrep.com/patent/US10769764B2

## CPC

- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 3/04: Context-preserving transformations, e.g. by using an importance map (panospheric to cylindrical image transformations G06T3/12)
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/11: Region-based segmentation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20212: Image combination
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A style of a digital image is transferred to another digital image of arbitrary resolution. A high-resolution (HR) content image is segmented into several low-resolution (LR) patches. The resolution of a style image is matched to have the same resolution as the LR content image patches. Style transfer is then performed on a patch-by-patch basis using, for example, a pair of feature transforms—whitening and coloring. The patch-by-patch style transfer process is then repeated at several increasing resolutions, or scale levels, of both the content and style images. The results of the style transfer at each scale level are incorporated into successive scale levels up to and including the original HR scale. As a result, style transfer can be performed with images having arbitrary resolutions to produce visually pleasing results with good spatial consistency.

## Claims

### Claim 1

In a digital medium environment for editing digital images, a computer-implemented method of transferring a style from one digital image to another digital image, the method comprising:
segmenting, by at least one processor of a computing device, an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring, by the at least one processor, a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling, by the at least one processor, the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining, by the at least one processor, the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting, by the at least one processor, the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring, by the at least one processor, the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing, by the at least one processor, display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.

### Claim 2 (depends on 1)

The method of claim 1, wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.

### Claim 3 (depends on 2)

The method of claim 2, wherein the whitening transform includes applying a first linear transformation to the content feature map, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of the covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.

### Claim 4 (depends on 3)

The method of claim 3, wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches, respectively.

### Claim 5 (depends on 1)

The method of claim 1, wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.

### Claim 6 (depends on 1)

The method of claim 1, further comprising repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second content scale is greater than the first content scale by a factor of two.

### Claim 8

A non-transitory computer readable medium having instructions encoded thereon that when executed by at least one processor of a computing device cause the at least one processor to perform a process of transferring a style from one digital image to another digital image, the process comprising:
segmenting, by the at least one processor, an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring, by the at least one processor, a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling, by the at least one processor, the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining, by the at least one processor, the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting, by the at least one processor, the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring, by the at least one processor, the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing, by the at least one processor, display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.

### Claim 9 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein the whitening transform includes applying a first linear transformation to a content feature map of each of the respective first and second plurality of patches, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of a covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.

### Claim 11 (depends on 10)

The non-transitory computer readable medium of claim 10, wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches.

### Claim 12 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.

### Claim 13 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the process further comprises repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.

### Claim 14

A system for transferring a style from one digital image to another digital image, the system comprising:
a storage; and
a processor operatively coupled to the storage, the processor configured to execute instructions stored in the storage that when executed cause the processor to carry out a process including
segmenting an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.

### Claim 15 (depends on 14)

The system of claim 14, wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.

### Claim 16 (depends on 15)

The system of claim 15, wherein the whitening transform includes applying a first linear transformation to a content feature map of each of the respective first and second plurality of patches, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of a covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.

### Claim 17 (depends on 16)

The system of claim 16, wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches, respectively.

### Claim 18 (depends on 14)

The system of claim 14, wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.

### Claim 19 (depends on 14)

The system of claim 14, wherein the process further includes repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.

### Claim 20 (depends on 14)

The system of claim 14, wherein the second content scale is greater than the first content scale by a factor of two.