# US 12,217,466 B2 · Portrait stylization framework to control the similarity between stylized portraits and original photo

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,466 B2
- Document ID: US12217466B2
- Status: Granted
- Kind: B2
- Application number: 17/519,711
- Filed: 2021-11-05
- Published: 2023-05-11
- Granted: 2025-02-04
- Earliest claimed: 2021-11-05
- Assignees: Lemon Inc.
- Applicants: Lemon Inc.
- Inventors: Jing Liu; Chunpong Lai; Guoxian Song; Linjie Luo
- Web page: https://patentgrep.com/patent/US12217466B2

## CPC

- G06T 9/002: {using neural networks}
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 11/60: Creating or editing images; Combining images with text
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods directed to controlling the similarity between stylized portraits and an original photo are described. In examples, an input image is received and encoded using a variational autoencoder to generate a latent vector. The latent vector may be blended with latent vectors that best represent a face in the original user portrait image. The resulting blended latent vector may be provided to a generative adversarial network (GAN) generator to generate a controlled stylized image. In examples, one or more layers of the stylized GAN generator may be swapped with one or more layers of the original GAN generator. Accordingly, a user can interactively determine how much stylization vs. personalization should be included in a resulting stylized portrait.

## Claims

### Claim 1

A method for generating a stylized image, the method comprising:
receiving an input image;
generating, using a first encoder, a first latent code based on the input image;
generating, using a second encoder, a second latent code based on the input image;
causing a user interface to be displayed;
receiving, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code;
blending the first latent code and the second latent code to obtain a blended latent code;
generating, by a generative adversarial network (GAN) generator, a stylized image based on the blended latent code; and
providing the stylized image as an output.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assembling the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generating the stylized image using the assembled GAN generator.

### Claim 3 (depends on 2)

The method of claim 2, wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first encoder is a PSP encoder.

### Claim 5 (depends on 1)

The method of claim 1, wherein the second encoder is a variational hierarchical autoencoder.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating the first latent code from a first multilayer perceptron; and
generating the second latent code from a second multilayer perceptron.

### Claim 7

A system, comprising:
one or more hardware processors configured by machine-readable instructions to:
receive an input image;
generate, using a first encoder, a first latent code based on the input image;
generate, using a second encoder, a second latent code based on the input image;
cause a user interface to be displayed;
receive, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code
blend the first latent code and the second latent code to obtain a blended latent code;
generate, by a generative adversarial network generator, a stylized image based on the blended latent code; and
provide the stylized image as an output.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more hardware processors are further configured by machine-readable instructions to:
receive a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assemble the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generate the stylized image using the assembled GAN generator.

### Claim 9 (depends on 8)

The system of claim 8, wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

### Claim 10 (depends on 7)

The system of claim 7, wherein the first encoder is a PSP encoder.

### Claim 11 (depends on 7)

The system of claim 7, wherein the second encoder is a variational hierarchical autoencoder.

### Claim 12 (depends on 7)

The system of claim 7, wherein the one or more hardware processors are further configured by machine-readable instructions to:
generate the first latent code from a first multilayer perceptron; and
generate the second latent code from a second multilayer perceptron.

### Claim 13

A non-transitory computer-readable storage medium comprising instructions, which when executed by one or more processors, cause the one or more processors to:
receive an input image;
generate, using a first encoder, a first latent code based on the input image;
generate, using a second encoder, a second latent code based on the input image;
cause a user interface to be displayed;
receive, via input to the user interface, a blending parameter indicating an amount to blend the first latent code with the second latent code
blend the first latent code and the second latent code to obtain a blended latent code;
generate, by a generative adversarial network generator, a stylized image based on the blended latent code; and
provide the stylized image as an output.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
receive a blending parameter indicating one or more layers of a first pre-trained GAN generator are to be used in the GAN generator;
assemble the GAN generator based on the blending parameter and the one or more layers of the pre-trained GAN generator; and
generate the stylized image using the assembled GAN generator.

### Claim 15 (depends on 14)

The non-transitory computer-readable storage medium of claim 14, wherein the GAN generator is a trained GAN generator trained via transfer learning from the first pre-trained GAN generator.

### Claim 16 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the second encoder is a variational hierarchical autoencoder.

### Claim 17 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:
generate the first latent code from a first multilayer perceptron; and
generate the second latent code from a second multilayer perceptron.

### Claim 18 (depends on 1)

The method of claim 1, wherein the causing a user interface to be displayed comprises causing a control to be displayed on the user interface, and wherein the control is interactable to allow a user to provide the blending parameter via the control.

### Claim 19 (depends on 18)

The method of claim 18, wherein the control is a slider.

### Claim 20 (depends on 18)

The method of claim 18, further comprising causing the stylized image to be displayed on the user interface.