Image generation with flexible sampler for diffusion modeling
AssigneeGenerate Biomedicines, Inc.
InventorsWujie Wang, John Ingraham
Life of this patent
Estimating the term…
Abstract
A system performs image generation with a flexible sampler used in a diffusion model. The system receives a set of sampling parameter(s) to tailor sampling. The sampling parameter(s) include a combination of a momentum sampling parameter, a mass sampling parameter, a friction sampling parameter, an equilibration sampling parameter, and an inverse temperature sampling parameter. The system modifies a sampling operation based on the sampling parameter(s). The system applies a diffusion model with the modified sampling operation to generate a synthetic image. The system may present the synthetic image on a graphical user interface.
Claims20
A computer-implemented method comprising:
- receiving, via a graphical user interface, a text prompt that specify features of a synthetic image to be generated;
- generating one or more embeddings for the text prompt in an embedding space;
- receiving a set of one or more sampling parameters that tailor sampling, wherein the set of one or more sampling parameters includes a combination of a momentum sampling parameter, a mass sampling parameter, a friction sampling parameter, an equilibration sampling parameter, and an inverse temperature sampling parameter;
- modifying a sampling operation based on the set of one or more sampling parameters to yield a first modified sampling operation;
- applying a diffusion model to generate the synthetic image, wherein applying the diffusion model comprises, in each sampling step of a plurality of sampling steps:
- denoising a prior sampled state of the synthetic image based on the one or more embeddings of the text prompt;
- and
- sampling a subsequent sampled state from the denoised prior sampled state by applying the first modified sampling operation,
- wherein the final sampled state is the synthetic image that satisfies the features of the text prompt;
- and
- providing, via the graphical user interface, the synthetic image for presentation.
A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the computer processor to:
- receive, via a graphical user interface, a text prompt that specify features of a synthetic image to be generated;
- generate one or more embeddings for the text prompt in an embedding space;
- receive a set of one or more sampling parameters that tailor sampling, wherein the set of one or more sampling parameters includes a combination of a momentum sampling parameter, a mass sampling parameter, a friction sampling parameter, an equilibration sampling parameter, and an inverse temperature sampling parameter;
- modify a sampling operation based on the set of one or more sampling parameters to yield a first modified sampling operation;
- apply a diffusion model to generate the synthetic image, wherein to apply the diffusion model comprises, in each sampling step of a plurality of sampling steps, to:
- denoise a prior sampled state of the synthetic image based on the one or more embeddings of the text prompt;
- and
- sample a subsequent sampled state from the denoised prior sampled state by applying the first modified sampling operation,
- wherein the final sampled state is the synthetic image that satisfies the features of the text prompt;
- and
- provide, via the graphical user interface, the synthetic image for presentation.
A computer-implemented method comprising:
- receiving, via a graphical user interface, a text prompt that specifies features of a structured data object to be generated;
- generating one or more embeddings for the text prompt in an embedding space;
- receiving a set of one or more sampling parameters that tailor sampling, wherein the set of one or more sampling parameters includes a combination of a momentum sampling parameter, a mass sampling parameter, a friction sampling parameter, an equilibration sampling parameter, and an inverse temperature sampling parameter;
- modifying a sampling operation based on the set of one or more sampling parameters to yield a first modified sampling operation;
- applying a diffusion model to generate the structured data object, wherein applying the diffusion model comprises, in each sampling step of a plurality of sampling steps:
- denoising a prior sampled state of the structured data object based on the one or more embeddings of the text prompt;
- and
- sampling a subsequent sampled state from the denoised prior sampled state by applying the first modified sampling operation,
- wherein the final sampled state is the structured data object that satisfies the features of the text prompt;
- rendering a visual representation of the structured data object;
- and
- providing, via the graphical user interface, the visual representation for presentation.
Dependencies are inferred from the claim text.