Method, device, and medium for generating transition videos with diffusion model

An electronic device, comprising:

• a memory and a processor;
• wherein the memory is configured to store one or more computer instructions which, when executed by the processor, cause the processor to:
• obtain a start frame and an end frame for a video, a first caption of the start frame, and a second caption of the end frame;
• generate a first latent noise in a latent space based on the start frame and a second latent noise in the latent space based on the end frame;
• generate third latent noises in the latent space corresponding to transition frames based on the first latent noise and the second latent noise;
• and
• generate, by utilizing a pre-trained image-to-video diffusion model, the transition frames based on the third latent noises, the start frame, the end frame, the first caption, and the second caption.

A non-transitory computer-readable medium comprising instructions stored thereon which, when executed by a processor, cause the processor to:

• obtain a start frame and an end frame for a video, a first caption of the start frame, and a second caption of the end frame;
• generate a first latent noise in a latent space based on the start frame and a second latent noise in the latent space based on the end frame;
• generate third latent noises in the latent space corresponding to transition frames based on the first latent noise and the second latent noise;
• and
• generate, by utilizing a pre-trained image-to-video diffusion model, the transition frames based on the third latent noises, the start frame, the end frame, the first caption, and the second caption.