# US 2023/0274535 A1 · User-guided image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274535 A1
- Document ID: US20230274535A1
- Status: Published application
- Kind: A1
- Application number: 17/680,906
- Filed: 2022-02-25
- Published: 2023-08-31
- Earliest claimed: 2022-02-25
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Yijun Li; Utkarsh Ojha; Richard Zhang; Jingwan Lu; Elya Shechtman; Alexei A. Efros
- Web page: https://patentgrep.com/patent/US20230274535A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An image generation system enables user input during the process of training a generative model to influence the model's ability to generate new images with desired visual features. A source generative model for a source domain is fine-tuned using training images in a target domain to provide an adapted generative model for the target domain. Interpretable factors are determined for the source generative model and the adapted generative model. A user interface is provided that enables a user to select one or more interpretable factors. The user-selected interpretable factor(s) are used to generate a user-adapted generative model, for instance, by using a loss function based on the user-selected interpretable factor(s). The user-adapted generative model can be used to create new images in the target domain.

## Claims

### Claim 1

One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
generating an adapted generative model by fine-tuning a source generative model using a set of training images;
determining a plurality of interpretable factors for the source generative model and the adapted generative model;
providing a user interface that presents user interface elements for selecting one or more of the interpretable factors;
receiving input indicative of a selection of an interpretable factor; and
generating a user-adapted generative model using a loss function based on the selected interpretable factor.

### Claim 2 (depends on 1)

The computer storage media of claim 1, wherein the plurality of interpretable factors are determined using an unsupervised method for discovering interpretable factors.

### Claim 3 (depends on 2)

The computer storage media of claim 2, wherein the interpretable factors for the source generative model are based on weights from a weight matrix of the source generative model, and wherein the interpretable factors for the adapted generative model are based on weights from a weight matrix of the adapted generative model.

### Claim 4 (depends on 1)

The computer storage media of claim 1, wherein the user interface presents a first image generated by the source generative model and a second image generated by the adapted generative model, the first image for a first interpretable factor for the source generative model and the second image for a corresponding first interpretable factor for the adapted generative model.

### Claim 5 (depends on 4)

The computer storage media of claim 4, wherein the input is indicative of a user selection associated with: the first image, the second image, and/or a user interface element associated with the first image and/or the second image.

### Claim 6 (depends on 4)

The computer storage media of claim 4, wherein the loss function minimizes a distance between the first interpretable factor for the source generation model and the corresponding first interpretable factor for the adapted generative model.

### Claim 7 (depends on 6)

The computer storage media of claim 6, wherein the distance comprises a cosine distance.

### Claim 8 (depends on 1)

The computer storage media of claim 1, wherein generating the user-adapted generative model comprises fine-tuning the adapted generative model using the loss function.

### Claim 9 (depends on 1)

The computer storage media of claim 1, wherein the operations further comprise:
generating a new image using the user-adapted generative model.

### Claim 10

A computerized method comprising:
receiving a source generative model for a source domain and a set of training images for a target domain;
adapting the source generative model using the set of training images to provide an adapted generative model for the target domain;
determining a plurality of interpretable factors for the source generative model and/or the adapted generative model;
receiving input regarding a user-selected interpretable factor from the plurality of interpretable factors;
generating a user-adapted generative model for the target domain based on the user-selected interpretable factor; and
using the user-adapted generative model to generate a new image in the target domain.

### Claim 11 (depends on 10)

The method of claim 10, wherein the plurality of interpretable factors are determined using an unsupervised method.

### Claim 12 (depends on 10)

The method of claim 10, wherein the input regarding the user-selected interpretable factor is received via a user interface that presents a visualization of at least a portion of the plurality of interpretable factors.

### Claim 13 (depends on 10)

The method of claim 10, wherein the user-adapted generative model is generated using a loss function that minimizes a distance between the user-selected interpretable factor for the source generative model and the user-adapted generative model.

### Claim 14 (depends on 13)

The method of claim 13, wherein the distance between the user-selected interpretable factor for the source generative model and the user-adapted generative model comprises a cosine distance.

### Claim 15 (depends on 13)

The method of claim 13, wherein generating the user-adapted generative model comprises fine-tuning the adapted generative model using the loss function.

### Claim 16

A computer system comprising:
a processor; and
a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
determining a plurality of interpretable factors for a first generative model and a corresponding plurality of interpretable factors for a second generative model adapted based on the first generative model;
receiving input regarding a user selection associated with a first interpretable factor for the first generative model and a corresponding first interpretable factor for the second generative model; and
generating a third generative model based on the user selection.

### Claim 17 (depends on 16)

The system of claim 16, wherein the third generative model is trained to minimize a loss between the first interpretable factor for the first generative model and a corresponding first interpretable factor for the third generative model.

### Claim 18 (depends on 17)

The system of claim 17, wherein the third generative model is generated by fine-tuning the second generative model.

### Claim 19 (depends on 18)

The system of claim 18, wherein the first interpretable factor for the first generative model corresponds to weights from a weight matrix of the first generative model and the corresponding first interpretable factor for the third generative model corresponds to weights from a weight matrix of the third generative model.

### Claim 20 (depends on 16)

The system of claim 16, wherein the input regarding the user selection is received via a user interface that presents a visualization of at least a portion of the plurality of interpretable factors for the first generative model and the corresponding plurality of interpretable factors for the second generative model.
