Single stream transformer for text-to-image/video synthesis

A system comprising:

• one or more memory devices;
• and
• one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
• receiving a text prompt to generate an image or video;
• generating, utilizing a text encoder, text tokens from the text prompt;
• generating combined tokens by combining the text tokens with noised tokens;
• generating, utilizing a single stream transformer comprising a self-attention layer and a multi-layer perceptron, denoised tokens by denoising the noised tokens in a manner that incorporates a context indicated by the text tokens and a token-level diffusion timestep embedding;
• and
• generating, utilizing a decoder, the image or the video from the denoised tokens.

A computer-implemented method comprising:

• receiving a text prompt to generate an image or video;
• generating, utilizing a text encoder, text tokens from the text prompt;
• generating combined tokens by combining the text tokens with noised tokens;
• generating, utilizing a diffusion transformer that does not include a cross-attention layer and modulation layers, denoised tokens by removing noise from the noised tokens in a manner that incorporates a context indicated by the text tokens;
• and
• generating, utilizing a decoder, the image or the video from the denoised tokens.