# US 12,217,395 B2 · Exemplar-based object appearance transfer driven by correspondence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,395 B2
- Document ID: US12217395B2
- Status: Granted
- Kind: B2
- Application number: 17/660,968
- Filed: 2022-04-27
- Published: 2023-11-02
- Granted: 2025-02-04
- Earliest claimed: 2022-04-27
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Sangryul Jeon; Zhifei Zhang; Zhe Lin; Scott Cohen; Zhihong Ding
- Web page: https://patentgrep.com/patent/US12217395B2

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/513: {Sparse representations}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20221: Image fusion; Image merging
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for image processing are configured. Embodiments of the present disclosure encode a content image and a style image using a machine learning model to obtain content features and style features, wherein the content image includes a first object having a first appearance attribute and the style image includes a second object having a second appearance attribute; align the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between a sparse set of pixels of the content image and corresponding pixels of the style image; and generate a hybrid image based on the sparse correspondence map, wherein the hybrid image depicts the first object having the second appearance attribute.

## Claims

### Claim 1

A method for image processing, comprising:
encoding a content image and a style image using a machine learning model to obtain content features and style features, wherein the content image includes a first object having a first appearance attribute and the style image includes a second object having a second appearance attribute;
computing a pixel-wise similarity score between a pixel from the content image and a pixel from the style image based on the content features and the style features;
including the pixel from the content image in a sparse set of pixels of the content image based on the pixel-wise similarity score;
aligning the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between the sparse set of pixels of the content image and corresponding pixels of the style image; and
generating a hybrid image based on the sparse correspondence map, wherein the hybrid image depicts the first object having the second appearance attribute.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating a first content feature map at a first scale and a second content feature map at a second scale, wherein the content features include the first content feature map and the second content feature map.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating a first style feature map at a first scale and a second style feature map at a second scale, wherein the style features include the first style feature map and the second style feature map.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
computing a first maximum pixel-wise similarity score for the pixel from the content image;
computing a second maximum pixel-wise similarity score for the pixel from the style image; and
computing a consistency score based on the first maximum pixel-wise similarity score and the second maximum pixel-wise similarity score, wherein the pixel from the content image is included in the sparse set of pixels of the content image based on the consistency score.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
identifying a layer index for a decoder of the machine learning model;
providing the content features as a first input to at least one layer of the decoder having a first index that is less than the layer index;
combining the sparse correspondence map with the style features to obtain aligned style features; and
providing the aligned style features as a second input to at least one subsequent layer of the decoder having a second index that is greater than the layer index, wherein the hybrid image comprises an output of the decoder.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
determining a desired level of detail to be transferred from the style image to the content image; and
selecting the layer index based on the desired level of detail.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
determining that the second object corresponds to the first object, wherein the alignment is based on the determination.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
identifying a first plurality of objects in the content image;
identifying a second plurality of objects in the style image; and
mapping the first plurality of objects to the second plurality of objects, wherein the alignment is based on the mapping.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
the second appearance attribute comprises color information, texture information, or a combination thereof.

### Claim 10

A method for training a machine learning model, comprising:
receiving a training set including a content image and a style image;
encoding the content image and the style image using a machine learning model to obtain content features and style features;
computing a pixel-wise similarity score between a pixel from the content image and a pixel from the style image based on the content features and the style features;
including the pixel from the content image in a sparse set of pixels of the content image based on the pixel-wise similarity score;
aligning the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between the sparse set of pixels of the content image and corresponding pixels of the style image;
generating a hybrid image based on the sparse correspondence map, wherein the hybrid image depicts an object from the content image and an appearance attribute from the style image; and
updating parameters of the machine learning model based on the hybrid image.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
generating a negative sample pair by randomly cropping the content image;
generating a positive sample pair by cropping the content image based on the pixel-wise similarity score; and
computing a correspondence contrastive loss based on the negative sample pair and the positive sample pair, wherein the parameters of the machine learning model are updated based on the correspondence contrastive loss.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
encoding the hybrid image to obtain hybrid features; and
computing a regularization loss based on the content features, the style features, and the hybrid features, wherein the parameters of the machine learning model are updated based on the regularization loss.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
generating a reconstructed content image based on the content features; and
computing a reconstruction loss based on the content image and the reconstructed content image, wherein the parameters of the machine learning model are updated based on the reconstruction loss.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
generating a reconstructed style image based on the style features, wherein the reconstruction loss is computed based on the style image and the reconstructed style image.

### Claim 15

An apparatus for image processing, comprising:
an image encoder configured to encode a content image and a style image using a machine learning model to obtain content features and style features, wherein the content image includes a first object having a first appearance attribute and the style image includes a second object having a second appearance attribute;
an alignment component configured to align the content features and the style features to obtain a sparse correspondence map that indicates a correspondence between a sparse set of pixels of the content image and corresponding pixels of the style image; and
an image decoder configured to generate a hybrid image based on the sparse correspondence map by identifying a layer index for the image encoder, providing the content features as a first input to at least one layer of the image decoder having a first index that is less than the layer index, combining the sparse correspondence map with the style features to obtain aligned style features, and providing the aligned style features as a second input to at least one subsequent layer of the image decoder having a second index that is greater than the layer index, wherein the hybrid image depicts the first object having the second appearance attribute.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein:
the image encoder comprises a multi-scale convolutional neural network (CNN).

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein:
the image decoder comprises a generative adversarial network (GAN).

### Claim 18 (depends on 15)

The apparatus of claim 15, further comprising:
a modulation encoder configured to project feature maps from the image encoder into an input vector space of the image decoder.
