Image Generation with Minimal Denoising Diffusion Steps
AssigneeGoogle Inc.
InventorsTingbo Hou, Yanwu Xu, Yang Zhao, Zhisheng Xiao
Life of this patent
Estimating the term…
Abstract
Provided is a one-step text-to-image generative model, which represents a fusion of GAN and diffusion model elements. In particular, despite the promising outcomes of prior diffusion GAN hybrid models, achieving one-step sampling and extending their utility to text-to-image generation remains a complex challenge. The present disclosure provides a number of innovative techniques to enhance diffusion GAN models, resulting in an ultra-fast text-to-image model capable of producing high-quality images in a single sampling step.
Claims20
A computer-implemented method to train machine learning models, the method comprising:
- obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters;
- instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as a generator model having the set of pre-trained model parameters;
- instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters;
- and
- finetuning, by the computing system, at least the generator model on a finetuning dataset, wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model.
A computer system comprising one or more computing devices, the computing system configured to perform operations to train image generation models, the operations comprising:
- obtaining a training example comprising a training image;
- performing one or more forward diffusion steps on the training example to generate a partially noised training example;
- performing an additional forward diffusion step on the partially noised training example to generate an additionally noised training example;
- processing the additionally noised training example with a generator model to generate fully de-noised prediction, wherein the generator model comprises a denoising diffusion model;
- performing one or more forward diffusion steps on the fully de-noised prediction to generate a partially re-noised prediction;
- processing the partially re-noised prediction with a discriminator model to generate a discriminator prediction;
- and
- updating one or more parameter values of at least the generator model based on a loss function, wherein the loss function comprises:
- a reconstruction loss term that generates a reconstruction loss value based on the training example and the fully de-noised prediction, and a GAN loss term that generates a GAN loss value based on the discriminator prediction.
One or more non-transitory computer-readable media that store a generator model that has been trained by performance of training operations, the training operations comprising:
- obtaining, by a computing system comprising one more computing devices, a pre-trained denoising diffusion model comprising a set of pre-trained model parameters;
- instantiating, by the computing system, a first instance of the pre-trained denoising diffusion model as the generator model having the set of pre-trained model parameters;
- instantiating, by the computing system, a second instance of the pre-trained denoising diffusion model as a discriminator model having the set of pre-trained model parameters;
- and
- finetuning, by the computing system, at least the generator model on a finetuning dataset, wherein finetuning, by the computing system, the generator model comprises modifying, by the computing system, the set of pre-trained model parameters of the generator model based on a generative adversarial network loss term that provides a loss value based on an output of the discriminator model.
Dependencies are inferred from the claim text.