Video generation with latent diffusion models

A computerized method for video generation corresponding to an input text, the method comprising:

• receiving the input text from a user;
• generating a plurality of key frames based on the input text using a latent diffusion model, wherein the latent diffusion model includes a shared two-dimensional convolution for the generation of the plurality of key frames, and wherein the latent diffusion model further includes, for each of the plurality of key frames, a respective adaptor comprising a plurality of learnable parameters for adjusting a mean and a variance of the key frame;
• interpolating the plurality of key frames;
• and
• outputting a video using the interpolated plurality of key frames.

A computing system for video generation corresponding to an input text, the computing system comprising:

• a processor and memory of a computing device, the processor being configured to execute a program using portions of the memory to:
• receive the input text from a user;
• generate a plurality of key frames based on the input text using a latent diffusion model, wherein the latent diffusion model includes:
• a shared two-dimensional convolution for the generation of the plurality of key frames;
• for each of the plurality of key frames that is to be generated, a respective adaptor comprising a plurality of learnable parameters for adjusting a mean and a variance of the key frame;
• and
• a directed temporal self-attention module for capturing temporal information among the plurality of key frames;
• interpolate the plurality of key frames by, for each pair of adjacent key frames, generating and inserting a transition frame between the pair of adjacent key frames, wherein the interpolation is performed by an interpolation model trained in latent space;
• and
• upsample the interpolated plurality of key frames using a super-resolution model;
• and
• output a video using the upsampled plurality of key frames.