Back to results
US 12,586,364 B2Granted
Single image concept encoder for personalization using a pretrained diffusion model
AssigneeAdobe Inc.
InventorsSaeid Motiian, Shabnam Ghadar
Life of this patent
Estimating the term…
Abstract
Systems and methods for image processing are provided. One aspect of the systems and methods includes identifying a style image including a target style. A style encoder network generates a style vector representing the target style based on the style image. The style encoder can be trained based on a style loss that encourages the network to match a desired style. A a diffusion model generates a synthetic image that includes the target style based on the style vector. The diffusion model is trained independently of the style encoder network.
Claims19
A method comprising:
- identifying a style image and a text prompt, wherein the style image indicates a target style and the text prompt indicates image content;
- generating a style vector representing the target style based on the style image using a style encoder network;
- generating a guidance embedding by inserting the style vector corresponding to the style image into an embedding of the text prompt in place of a part of the text prompt, wherein the guidance embedding represents the target style from the style image in an embedding space for guidance of a diffusion model;
- and
- generating, using the diffusion model, a synthetic image by denoising noisy image features based on the guidance embedding using the diffusion model, wherein the synthetic image depicts the image content from the text prompt with the target style from the style image.
2 more elements
A method comprising:
- identifying a plurality of training images including a target style;
- computing an optimized style vector based on the plurality of training images;
- identifying a training image that includes the target style;
- encoding the training image using a style encoder network to obtain a style vector representing the target style in a latent space for guidance of a diffusion model, wherein the style vector represents the target style from the training image in an embedding space for guidance of the diffusion model;
- computing a style loss based on the training image by comparing the style vector to the optimized style vector;
- training the style encoder network by updating parameters of the style encoder network based on the style loss;
- and
- inserting an output of the trained style encoder network into an embedding of a text prompt in place of a part of the text prompt to obtain a guidance embedding for the diffusion model.
5 more elements
An apparatus comprising:
- a processor;
- and
- a memory including instructions executable by the processor to:
- identify a style image and a text prompt, wherein the style image indicates a target style and the text prompt indicates image content;
- generate a style vector representing the target style based on the style image using a style encoder network;
- generate a guidance embedding by inserting the style vector corresponding to the style image into an embedding of the text prompt in place of a part of the text prompt, wherein the guidance embedding represents the target style from the style image in an embedding space for guidance of a diffusion model;
- and
- generate, using the diffusion model, a synthetic image by denoising noisy image features based on the guidance embedding using a diffusion model, wherein the synthetic image depicts the image content from the text prompt with the target style from the style image.
5 more elements
Dependencies are inferred from the claim text.