Back to results
US 2024/0185588 A1Published application
Fine-tuning and controlling diffusion models
AssigneeAdobe Inc.
InventorsNupur Kumari, Richard Zhang, Junyan Zhu, Elya Shechtman
Life of this patent
Estimating the term…
Abstract
Systems and methods for fine-tuning diffusion models are described. Embodiments of the present disclosure obtain an input text indicating an element to be included in an image; generate a synthetic image depicting the element based on the input text using a diffusion model trained by comparing synthetic images depicting the element to training images depicting elements similar to the element and updating selected parameters corresponding to an attention layer of the diffusion model based on the comparison.
Claims20
A method comprising:
- obtaining an input text indicating an element to be included in an image;
- and
- generating a synthetic image depicting the element based on the input text using a diffusion model, wherein the diffusion model is trained by comparing synthetic images depicting the element to training images depicting elements similar to the element and updating selected parameters corresponding to an attention layer of the diffusion model based on the comparison.
A method comprising:
- identifying a diffusion model trained on a first training set during a first training phase;
- selecting a first subset of parameters of the diffusion model corresponding to an attention block and a second subset of parameters of the diffusion model outside of the attention block;
- and
- training the diffusion model to generate a synthetic image based on text condition features by fine-tuning the first subset of parameters of the diffusion model based on a second training set different from the first training set during a second training phase, wherein the second subset of parameters of the diffusion model are held fixed during the second training phase.
1 more element
An apparatus comprising:
- one or more processors;
- one or more memories including instructions executable by the one or more processors to:
- generate text condition features from an input text using a text transformer model;
- select images related to the input text;
- and
- generate synthetic images based on the text condition features using a diffusion model, wherein the diffusion model is trained by updating selected parameters corresponding to one or more attention layers of the diffusion model.
3 more elements
Dependencies are inferred from the claim text.