High-resolution video generation using image diffusion models
AssigneeNVIDIA Corporation
InventorsKarsten Julian Kreis, Robin Rombach, Andreas Blattmann, Seung Wook Kim, Huan Ling, Sanja Fidler, Tim Dockhorn
Life of this patent
Estimating the term…
Abstract
In various examples, systems and methods are disclosed relating to aligning images into frames of a first video using at least one first temporal attention layer of a neural network model. The first video has a first spatial resolution. A second video having a second spatial resolution is generated by up-sampling the first video using at least one second temporal attention layer of an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.
Claims18
A system, comprising one or more processors to:
- align a plurality of images into frames of a first video using a first neural network model;
- and
- generate a second video by up-sampling the first video using a second neural network model, wherein the second video has a resolution that is higher than a resolution of the first video, wherein the first neural network is updated according to one or more temporal incoherencies in mapping latent encodings from a latent space to an image space, wherein
- the first neural network model comprises a first diffusion model and a second diffusion model;
- the first diffusion model is to generate a third video;
- and
- the second diffusion model is to generate the first video by generating at least one frame between two consecutive frames of the third video.
A datacenter, comprising one or more processors to:
- align a plurality of images into frames of a first video using a first neural network model;
- and
- generate a second video by up-sampling the first video using a second neural network model, wherein the second video has a resolution that is higher than a resolution of the first video, wherein the first neural network is updated according to one or more temporal incoherencies in mapping latent encodings from a latent space to an image space, wherein
- the first neural network model comprises a first diffusion model and a second diffusion model;
- the first diffusion model is to generate a third video;
- and
- the second diffusion model is to generate the first video by generating at least one frame between two consecutive frames of the third video.
A method, comprising:
- align, by one or more processors, a plurality of images into frames of a first video using a first neural network model;
- and
- generate, by one or more processors, a second video by up-sampling the first video using a second neural network model, wherein the second video has a resolution that is higher than a resolution of the first video, wherein the first neural network is updated according to one or more temporal incoherencies in mapping latent encodings from a latent space to an image space, wherein
- the first neural network model comprises a first diffusion model and a second diffusion model;
- the first diffusion model is to generate a third video;
- and
- the second diffusion model is to generate the first video by generating at least one frame between two consecutive frames of the third video.
Dependencies are inferred from the claim text.