# US 2026/0073580 A1 · Single stream transformer for text-to-image/video synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073580 A1
- Document ID: US20260073580A1
- Status: Published application
- Kind: A1
- Application number: 18/930,672
- Filed: 2024-10-29
- Published: 2026-03-12
- Earliest claimed: 2024-09-11
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Kai Zhang; Jianming Zhang; Sai Bi; Zexiang Xu; Hao Tan; Wei-An Lin
- Web page: https://patentgrep.com/patent/US20260073580A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/094: Adversarial learning
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates an image or a video from a text prompt. For example, the disclosed systems receive a text prompt and generates text tokens from the text prompt. Moreover, the disclosed systems generate combined tokens by combining the text tokens with noised tokens. Further, the disclosed systems generate denoised tokens by removing noise from noised tokens in a manner that incorporates a context indicated by the text tokens and further generates an image or video from the denoised tokens.

## Claims

### Claim 1

A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a text prompt to generate an image or video;
generating, utilizing a text encoder, text tokens from the text prompt;
generating combined tokens by combining the text tokens with noised tokens;
generating, utilizing a single stream transformer comprising a self-attention layer and a multi-layer perceptron to process the combined tokens, denoised tokens by removing noise from the noised tokens in a manner that incorporates a context indicated by the text tokens; and
generating, utilizing a decoder, the image or the video from the denoised tokens.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the combined tokens comprises:
generating a token-level diffusion timestep embedding; and
adding the token-level diffusion timestep embedding to the noised tokens to generate the combined tokens.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the combined tokens comprises:
generating position encodings for the image or the video; and
adding the position encodings for the image or the video to the noised tokens to generate the combined tokens.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the image or the video further comprises:
generating, utilizing the single stream transformer to process the combined tokens comprising text tokens, the noised tokens, a token-level diffusion timestep embedding, and position encodings, denoised tokens by removing noise from the noised tokens according to the text tokens, the token-level diffusion timestep embedding, and the position encodings;
discarding the text tokens; and
generating, utilizing the decoder to process the denoised tokens, the image or the video.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein utilizing the single stream transformer comprises utilizing a transformer that does not have conditioning inputs to denoise the noised tokens for the text prompt by:
generating, utilizing the self-attention layer to process the noised tokens, a self-attention layer output; and
combining the self-attention layer output with the noised tokens to generate a combined self-attention layer output.

### Claim 6 (depends on 5)

The non-transitory computer-readable medium of claim 5, further comprising:
generating, utilizing the multi-layer perceptron, a multi-layer perceptron output from the combined self-attention layer output; and
combining the multi-layer perceptron output with the combined self-attention layer output to generate the denoised tokens.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising:
generating, utilizing a transformer block of the single stream transformer, intermediate denoised tokens from the noised tokens;
generating, utilizing an additional transformer block of the single stream transformer, the denoised tokens from the intermediate denoised tokens; and
generating, utilizing the decoder, the image or the video from the denoised tokens.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising:
receiving, in addition to the text prompt, a visual prompt that includes a digital image;
generating, utilizing an encoder of a two-dimensional variational autoencoder, visual tokens from the digital image;
generating the combined tokens by combining the text tokens, the visual tokens, and the noised tokens;
generating, utilizing the single stream transformer to process the combined tokens, denoised tokens by removing the noise from the noised tokens in a manner that indicates the text tokens and the visual tokens; and
generating, utilizing the decoder, the video from the denoised tokens.

### Claim 9 (depends on 1)

The non-transitory computer-readable medium of claim 1, further comprising:
receiving, in addition to the text prompt, a visual prompt that includes a first digital image and a second digital image;
generating, utilizing an encoder of a two-dimensional variational autoencoder, a first set of visual tokens for the first digital image and a second set of visual tokens for the second digital image;
generating the combined tokens by combining the text tokens, the first set of visual tokens, the second set of visual tokens, and the noised tokens;
generating, utilizing the single stream transformer to process the combined tokens, denoised tokens by removing the noise from the noised tokens in a manner that indicates the text tokens and the first set of visual tokens and the second set of visual tokens; and
generating, utilizing the decoder, the video from the denoised tokens.

### Claim 10

A system comprising:
one or more memory devices; and
one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
receiving a text prompt to generate an image or video;
generating, utilizing a text encoder, text tokens from the text prompt;
generating combined tokens by combining the text tokens with noised tokens;
generating, utilizing a single stream transformer comprising a self-attention layer and a multi-layer perceptron, denoised tokens by denoising the noised tokens in a manner that incorporates a context indicated by the text tokens and a token-level diffusion timestep embedding; and
generating, utilizing a decoder, the image or the video from the denoised tokens.

### Claim 11 (depends on 10)

The system of claim 10, wherein the operations further comprise:
generating, utilizing a first transformer block of the single stream transformer, intermediate denoised tokens from processing the noised tokens and the token-level diffusion timestep embedding for the first transformer block; and
generating, utilizing a second transformer block of the single stream transformer, the denoised tokens from processing the intermediate denoised tokens and an additional token-level diffusion timestep embedding for the second transformer block.

### Claim 12 (depends on 10)

The system of claim 10, wherein generating the combined tokens comprises:
generating position encodings comprising at least one of a token-level diffusion timestep, a pixel location, a video frame timestamp, or a camera pose; and
adding the position encodings to the noised tokens to generate the combined tokens.

### Claim 13 (depends on 10)

The system of claim 10, wherein generating the image comprises generating, utilizing the decoder, the image from the denoised tokens according to position encodings indicating a camera pose, pixel locations, and a description of the text prompt.

### Claim 14 (depends on 10)

The system of claim 10, wherein generating the video comprises:
receiving, in addition to the text prompt, a visual prompt that includes a digital image;
generating the combined tokens by combining the text tokens, visual tokens generated from the digital image, and the noised tokens;
generating, utilizing the single stream transformer to process the combined tokens, denoised tokens by removing noise from the noised tokens in a manner that incorporates content indicated by the text tokens and the visual tokens; and
generating, utilizing the decoder, the video from the denoised tokens according to the text prompt and position encodings indicating pixel locations, video frame timestamps, and camera poses.

### Claim 15 (depends on 14)

The system of claim 14, wherein the single stream transformer consists of the self-attention layer and the multi-layer perceptron.

### Claim 16

A computer-implemented method comprising:
receiving a text prompt to generate an image or video;
generating, utilizing a text encoder, text tokens from the text prompt;
generating combined tokens by combining the text tokens with noised tokens;
generating, utilizing a diffusion transformer that does not include a cross-attention layer and modulation layers, denoised tokens by removing noise from the noised tokens in a manner that incorporates a context indicated by the text tokens; and
generating, utilizing a decoder, the image or the video from the denoised tokens.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16, wherein generating the denoised tokens comprises:
generating a first token-level diffusion timestep embedding for a first transformer block of the diffusion transformer; and
generating, utilizing the first transformer block of the diffusion transformer, a first intermediate denoised tokens by denoising the noised tokens in a manner indicated by the first token-level diffusion timestep embedding.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, further comprising:
generating a second token-level diffusion timestep embedding for a second transformer block of the diffusion transformer;
generating, utilizing the second transformer block of the diffusion transformer, a second intermediate denoised tokens by denoising the first intermediate denoised tokens in a manner indicated by the second token-level diffusion timestep embedding; and
generating, utilizing a third transformer block of the diffusion transformer, the denoised tokens by denoising the second intermediate denoised tokens in a manner indicated by a third token-level diffusion timestep embedding.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16, wherein utilizing the diffusion transformer comprises utilizing a single stream transformer that comprises a self-attention layer and a multi-layer perceptron to denoise the noised tokens by:
generating, utilizing a first transformer block of the self-attention layer to process the noised tokens, a self-attention layer output; and
combining the self-attention layer output with the noised tokens to generate a combined self-attention layer output.

### Claim 20 (depends on 19)

The computer-implemented method of claim 19, further comprising:
generating, utilizing the multi-layer perceptron, a multi-layer perceptron output from the combined self-attention layer output; and
combining the multi-layer perceptron output with the combined self-attention layer output to generate the denoised tokens.
