US 11,893,713 B1 · Augmented diffusion inversion using latent trajectory optimization

Abstract

Augmented Denoising Diffusion Implicit Models (“DDIMs”) using a latent trajectory optimization process can be used for image generation and manipulation using text input and one or more source images to create an output image. Noise bias and textual bias inherent in the model representing the image and text input is corrected by correcting trajectories previously determined by the model at each step of a diffusion inversion process by iterating multiple starts the trajectories to find determine augmented trajectories that minimizes loss at each step. The trajectories can be used to determine an augmented noise vector, enabling use of an augmented DDIM and resulting in more accurate, stable, and responsive text-based image manipulation.

Claims20

Dependent claims

A method of automated image processing, comprising:

receiving an image;
defining a diffusion model of the image;
generating a noise vector for the diffusion model by deterministically applying noise to the image;
encoding the image with a bias correction variable by replacing an empty textual embedding of the diffusion model with a conditional textual embedding to generate an augmented diffusion model;
generating an augmented noise vector using the noise vector, the augmented diffusion model, and a latent trajectory optimization process to determine one or more diffusion trajectories for the augmented noise vector;
and
automatically generating an output image based on the augmented noise vector.

4 more elements

A non-transitory computer readable storage medium comprising instructions, that when executed by one or more processors of a computing system, cause the computing system to:

receive an image;
define a diffusion model of the image;
generate a noise vector for the diffusion model by deterministically applying noise to the image;
encode the image with a bias correction variable by replacing an empty textual embedding of the diffusion model with a conditional textual embedding to generate an augmented diffusion model;
generate an augmented noise vector using the noise vector, the augmented diffusion model, and a latent trajectory optimization process to determine one or more diffusion trajectories for the augmented noise vector;
and
automatically generate an output image based on the augmented noise vector.

4 more elements

Dependencies are inferred from the claim text.

Augmented diffusion inversion using latent trajectory optimization

Life of this patent

Augmented diffusion inversion using latent trajectory optimization

Life of this patent

Abstract

Claims20