Dual-vae for more efficient and effective diffusion model training

A system comprising:

• one or more memory devices;
• and
• one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
• utilizing a trained dual-variational autoencoder model to train a diffusion transformer model by:
• generating denoised image tokens by denoising, utilizing the diffusion transformer model, image tokens to which noise has been added, the image tokens being generated by a two-dimensional variational autoencoder from a frame of a sequence of frames of a video;
• modifying parameters of the diffusion transformer model based on a comparison of the denoised image tokens and the image tokens;
• generating denoised motion tokens by denoising, utilizing the diffusion transformer model, motion tokens to which noise has been added, the motion tokens being generated by a three-dimensional variational autoencoder from the sequence of frames;
• and
• refining the modified parameters of the diffusion transformer model based on a comparison of the denoised motion tokens and the motion tokens.

A computer-implemented method comprising:

• receiving, from a client device, a media generation request comprising one or more of a text prompt or an image prompt;
• generating, utilizing a diffusion transformer model, denoised tokens from noised tokens generated from the media generation request;
• and
• generating, utilizing a decoder of a trained dual-variational autoencoder model, media from the denoised tokens, the trained dual-variational autoencoder model comprising a two-dimensional variational autoencoder that decodes digital images and a three-dimensional variational autoencoder that decodes motion frames.