# US 2024/0005574 A1 · Applying object-aware style transfer to digital images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005574 A1
- Document ID: US20240005574A1
- Status: Published application
- Kind: A1
- Application number: 17/810,392
- Filed: 2022-07-01
- Published: 2024-01-04
- Earliest claimed: 2022-07-01
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Zhifei Zhang; Zhe Lin; Scott Cohen; Darshan Prasad; Zhihong Ding
- Web page: https://patentgrep.com/patent/US20240005574A1

## CPC

- G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 2207/10024: Color image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for transferring global style features between digital images utilizing one or more machine learning models or neural networks. In particular, in one or more embodiments, the disclosed systems receive a request to transfer a global style from a source digital image to a target digital image, identify at least one target object within the target digital image, and transfer the global style from the source digital image to the target digital image while maintaining an object style of the at least one target object.

## Claims

### Claim 1

A non-transitory computer-readable medium storing executable instructions, which when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a request to transfer a global style from a source digital image to a target digital image;
identifying, in response to receiving the request to transfer, at least one target object within the target digital image; and
transferring, utilizing a style transfer neural network, the global style from the source digital image to the target digital image while maintaining an object style of the at least one target object in the source digital image to generate a modified digital image.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein identifying the at least one target object within the target digital image comprises utilizing an object detection machine learning model to identify at least one foreground object within the target digital image.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising extracting the at least one target object from within the target digital image utilizing a segmentation model.

### Claim 4 (depends on 3)

The non-transitory computer-readable medium of claim 3, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating, in response to extracting the at least one target object from within the target digital image, a content fill for a hole corresponding to the at least one target object; and
filling the hole corresponding to the at least one target object with the content fill.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising:
identifying, in response to receiving the request to transfer, at least one source object within the source digital image;
extracting the at least one source object from within the source digital image to generate an intermediate source digital image; and
generating the modified digital image by transferring the global style from the intermediate source digital image to the target digital image.

### Claim 6 (depends on 3)

The non-transitory computer-readable medium of claim 3, wherein generating the modified digital image further comprises reinserting the at least one target object into the target digital image in response to transferring the global style.

### Claim 7 (depends on 6)

The non-transitory computer-readable medium of claim 6, further comprising instructions, which when executed by the at least one processor, cause the at least one processor to perform operations comprising harmonizing the reinserted at least one target object with a background portion of the target digital image adjacent to the at least one target object.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image;
extracting, utilizing the encoder neural network, a spatial code from the target digital image corresponding to a geometric layout of the target digital image; and
generating, utilizing a generator neural network, the modified digital image by combining the global code of the source digital image with the spatial code of the target digital image.

### Claim 9

A computer-implemented method comprising:
receiving a request to transfer a global style from a source digital image to a target digital image;
extracting, in response to receiving the request to transfer, at least one target object from within the target digital image to generate a first intermediate digital image;
transferring, utilizing a style transfer neural network, the global style from the source digital image to the first intermediate digital image to generate a second intermediate digital image; and
inserting the at least one target object into the second intermediate digital image to generate a modified digital image.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the global style comprises a landscape texture within the source digital image.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the request to transfer comprises a selection of the at least one target object for extraction from within the target digital image.

### Claim 12 (depends on 9)

The computer-implemented method of claim 9, further comprising utilizing a segmentation model to extract the at least one target object from within the target digital image.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9, wherein generating the first intermediate digital image further comprises generating, utilizing a content aware fill machine learning model, a content fill for a hole corresponding to the at least one target object.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9, wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image;
extracting, utilizing the encoder neural network, a spatial code from the first intermediate digital image corresponding to a geometric layout of the first intermediate digital image; and
combining, utilizing a generator neural network, the global code with the spatial code to generate the second intermediate digital image.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9, wherein generating the modified digital image further comprises harmonizing the inserted at least one target object with a background portion of the second intermediate digital image adjacent to the at least one target object.

### Claim 16

A system comprising:
one or more memory devices comprising a source digital image, a target digital image, a segmentation model, and a style transfer neural network; and
one or more processors configured to cause the system to:
extract, utilizing the segmentation model, at least one target object from the target digital image to generate a first intermediate digital image;
transfer, utilizing the style transfer neural network, a global style from the source digital image to the first intermediate digital image to generate a second intermediate digital image; and
insert the at least one target object into the second intermediate digital image to generate a modified digital image.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more processors are further configured to cause the system to identify the at least one target object to be extracted from the target digital image utilizing an object detection machine learning model.

### Claim 18 (depends on 16)

The system of claim 16, wherein the one or more processors are further configured to cause the system to:
identify at least one source object in the source digital image, the at least one source object comprising a different style than the global style of the source digital image;
modify the source digital image by extracting, utilizing the segmentation model, the at least one source object from within the source digital image and generating a content fill for a hole corresponding to the at least one source object; and
in response to modifying the source digital image, transfer the global style from the source digital image to the first intermediate digital image to generate the modified digital image.

### Claim 19 (depends on 16)

The system of claim 16, wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image;
extracting, utilizing the encoder neural network, a spatial code from the target digital image corresponding to a geometric layout of the target digital image; and
generating, utilizing a generator neural network, the modified digital image by combining the global code of the source digital image with the spatial code of the target digital image.

### Claim 20 (depends on 16)

The system of claim 16, wherein the one or more processors are further configured to cause the system to generate the modified digital image by harmonizing, utilizing a harmonization neural network, the inserted at least one target object with a background of the second intermediate digital image adjacent to the at least one target object.
