# US 2025/0166664 A1 · Image Diffusion Framework for Text-Guided Video Editing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166664 A1
- Document ID: US20250166664A1
- Status: Published application
- Kind: A1
- Application number: 18/955,385
- Filed: 2024-11-21
- Published: 2025-05-22
- Earliest claimed: 2023-11-21
- Assignees: University of Central Florida Research Foundation, Inc.
- Applicants: University of Central Florida Research Foundation, Inc.
- Inventors: Nazmul Karim; Nazanin Rahnavard; Umar Khalid; Chen Chen
- Web page: https://patentgrep.com/patent/US20250166664A1

## CPC

- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The invention provides a method for adapting a text-to-image (T2I) diffusion model for video editing by using spectral decomposition to achieve controlled spectral shifts in the model's weights. This adaptation involves maintaining constant singular vectors while selectively adjusting singular values in response to a text prompt. A spectral shift regularizer constrains adjustments, particularly limiting changes to larger singular values to ensure minimal deviation from the original model's structure. This approach allows efficient, prompt-driven video editing by modifying specific elements according to the prompt while preserving the original video context. By focusing on selective spectral adjustments, the method reduces adaptation time and computational demands, making it suitable for real-time and resource-sensitive applications, such as dynamic video editing for streaming services.

## Claims

### Claim 1

A method for adapting a text-to-image (T2I) diffusion model to edit video content based on a text prompt, the method comprising:
a. receiving, by at least one processor, a video comprising a plurality of frames and a text prompt specifying modifications to visual elements in the video;
b. performing spectral decomposition on at least one weight matrix of the pre-trained T2I diffusion model to separate each matrix into a set of singular values and corresponding singular vectors;
c. generating a spectral shift parameter matrix by selectively adjusting only the singular values based on the text prompt, while maintaining the singular vectors unmodified;
d. applying a spectral shift regularizer to the spectral shift parameter matrix, wherein the spectral shift regularizer imposes more restricted adjustments to singular values with larger magnitudes and allows comparatively relaxed adjustments to singular values with smaller magnitudes;
e. adapting the pre-trained T2I diffusion model by incorporating the spectral shift parameter matrix, thereby creating an adapted model configured to modify specific visual elements within the video according to the text prompt; and
f. outputting an edited video in which the visual elements specified by the text prompt are modified in the plurality of frames, while preserving non-targeted visual elements and maintaining temporal coherence across frames.

### Claim 2 (depends on 1)

The method of claim 1, wherein the spectral decomposition is performed on query matrices within attention layers of the T2I diffusion model, and each query matrix is decomposed into singular values and singular vectors that are adjusted exclusively based on the spectral shift parameter matrix.

### Claim 3 (depends on 1)

The method of claim 1, further comprising calculating each adjustment in the spectral shift parameter matrix as a function of the text prompt, wherein the adjustment selectively increases or decreases each singular value to correspond with prompt-specified changes in visual content.

### Claim 4 (depends on 1)

The method of claim 1, wherein the spectral shift regularizer is configured to apply a rectified linear unit (ReLU) activation function to each adjustment in the spectral shift parameter matrix, the ReLU activation function limiting the adjustment of each singular value to a predetermined maximum value.

### Claim 5 (depends on 1)

The method of claim 1, further comprising initializing a temporal embedding across the plurality of frames in the video, wherein the temporal embedding is configured to enforce frame-to-frame coherence as the spectral shift parameter matrix propagates adjustments throughout the adapted model.

### Claim 6 (depends on 1)

The method of claim 1, wherein the adapted model is configured to maintain non-targeted regions of the video unchanged by limiting spectral shift adjustments to regions or features specified in the text prompt.

### Claim 7 (depends on 1)

The method of claim 1, further comprising generating cross-attention maps based on the text prompt to identify target regions within the latent representations of each frame, wherein the cross-attention maps serve to direct spectral shift adjustments to prompt-relevant areas within the video.

### Claim 8 (depends on 1)

The method of claim 1, wherein the spectral shift parameter matrix is iteratively fine-tuned over a set number of epochs to optimize alignment of the adapted model's output with the text prompt, wherein each epoch updates the spectral shift adjustments according to differences between the model output and prompt-specified content.

### Claim 9 (depends on 1)

The method of claim 1, further comprising generating an edited video output by processing the adapted model's latent representations of the video frames through a diffusion process, wherein the diffusion process applies adjustments from the spectral shift parameter matrix to produce frames aligned with the text prompt.

### Claim 10 (depends on 1)

The method of claim 1, wherein the adapted model employs classifier-free guidance to balance between original video content and the prompt-directed edits, and the classifier-free guidance operates by scaling the influence of prompt-driven modifications within the spectral shift parameter matrix.

### Claim 11

A method for adapting a text-to-image (T2I) diffusion model for text-driven video editing, the method comprising:
a. receiving, by at least one processor, an input video comprising a plurality of frames and an editing prompt;
b. encoding each frame of the input video to obtain clean latent representations, wherein the encoding reduces spatial redundancy and prepares the frames for diffusion-based processing;
c. initializing, by the at least one processor, an inflated text-to-video (T2V) model from the T2I diffusion model, wherein the T2V model includes temporal layers that manage frame-to-frame coherence and spatial consistency during the editing process;
d. performing spectral decomposition on each attention layer's query matrices in the T2V model to separate each matrix into singular values and singular vectors, enabling controlled adaptation of model weights;
e. generating a spectral shift parameter matrix by calculating differential adjustments to the singular values without modifying the singular vectors, where the spectral shift parameter matrix applies a rectified linear unit (ReLU) function to constrain adjustments;
f. applying a spectral shift regularizer to selectively adjust each singular value based on its magnitude, wherein larger singular values are constrained more tightly than smaller singular values to limit deviations in model behavior;
g. fine-tuning the T2V model by iteratively optimizing the spectral shift parameter matrix across frames to align the model output with the editing prompt; and
h. outputting an edited video in which the changes conform to the prompt specifications while preserving temporal coherence and maintaining the original video's background.

### Claim 12 (depends on 11)

The method of claim 11, wherein the fine-tuning further comprises sampling a noise embedding at each time step for each frame, wherein the noise embedding is used in a forward diffusion process to gradually increase noise, producing a set of progressively noised latent frames.

### Claim 13 (depends on 11)

The method of claim 11, wherein the spectral shift regularizer dynamically scales adjustments in inverse proportion to the magnitude of each singular value, thereby facilitating finer adjustments for minor components and tighter control over major components.

### Claim 14 (depends on 11)

The method of claim 11, wherein the input video is encoded as a sequence of temporally aligned latents, and the editing prompt guides the adjustments to specific regions or objects within the video frames.

### Claim 15 (depends on 11)

The method of claim 11, further comprising using a classifier-free guidance scale (scfg) hyperparameter to balance between the original content and the prompt-directed edits, wherein an scfg value of 1 preserves the original content and higher values intensify the influence of the prompt on the final video output.

### Claim 16 (depends on 11)

The method of claim 11, further comprising applying Deterministic Denoising Diffusion Implicit Model (DDIM) inversion to decode the noised latent frames, granularly reconstructing the frames according to the adjusted singular values guided by the prompt.

### Claim 17 (depends on 11)

The method of claim 11, wherein the method performs iterative steps to minimize the noise discrepancy between the model's noise estimates and the actual frame noise, updating the model parameters to improve prompt-specific alignment.

### Claim 18 (depends on 11)

The method of claim 11, wherein frame-wise consistency is maintained by initializing a temporal embedding across frames, allowing the model to retain spatial and temporal coherence as adjustments propagate through each frame.

### Claim 19 (depends on 11)

The method of claim 11, wherein the editing prompt is used to modify localized features in the video, including color, object identity, or movement direction, while preserving the continuity and context of non-targeted areas in each frame.

### Claim 20 (depends on 11)

The method of claim 11, wherein the T2V model is fine-tuned using a gradient-based optimization process that calculates a prompt-specific loss to iteratively refine the spectral shift parameters across a specified number of time steps.

### Claim 21

A method for zero-shot video generation from a text prompt using a pre-trained text-to-image (T2I) diffusion model, the method comprising:
a. receiving, by at least one processor, a text prompt describing the desired video content, wherein the prompt specifies attributes of objects and actions within a multi-frame sequence;
b. initializing a pre-trained T2I model with temporal modifications, producing a T2V model that includes frame-wise attention mechanisms adapted for video generation;
c. applying frame attention to generate temporally coherent frames based on the prompt without requiring task-specific fine-tuning, wherein the frame attention uses fixed keys and values derived from initial frames' latent representations;
d. employing sparse-causal cross-frame attention to link each frame to the immediate preceding frames, allowing temporal coherence while minimizing computational complexity;
e. utilizing spatio-temporal attention to create an attention map that spans all frames within the sequence, ensuring spatial continuity of objects and maintaining alignment across frames; and
f. generating, by the at least one processor, a multi-frame video that reflects the prompt-specified attributes in a temporally consistent manner, forming a coherent video sequence from the generated frames.

### Claim 22 (depends on 21)

The method of claim 21, wherein the cross-frame attention mechanism includes causal masking, ensuring that each frame only references prior frames to maintain causal consistency.

### Claim 23 (depends on 21)

The method of claim 21, wherein frame attention is limited to a subset of frames designated as anchor frames, from which keys and values propagate through subsequent frames to maintain temporal structure.

### Claim 24 (depends on 21)

The method of claim 21, further comprising encoding the prompt with a Contrastive Language-Image Pre-training (CLIP) text encoder, producing prompt embeddings that serve as input conditions for frame generation.

### Claim 25 (depends on 21)

The method of claim 21, wherein sparse-causal attention only references frames within a defined temporal window, reducing computation while maintaining temporal coherence for objects that persist across multiple frames.

### Claim 26 (depends on 21)

The method of claim 21, further comprising calculating spectral shift adjustments for singular values associated with attention weights, using a ReLU function to bound adjustments within a pre-specified range, thus ensuring stable generation across frames.

### Claim 27 (depends on 21)

The method of claim 21, wherein temporal embeddings are initialized for each frame sequence to enhance zero-shot generation capabilities by capturing the prompt-specific motion dynamics across frames.

### Claim 28 (depends on 21)

The method of claim 21, wherein the T2V model is configured with tunable parameters adjusted at runtime to emphasize specific aspects of the prompt, including object placement and motion direction.

### Claim 29 (depends on 21)

The method of claim 21, further comprising applying Deterministic Denoising Diffusion Implicit Model (DDIM) inversion to translate noisy latent representations into temporally synchronized frames based on the encoded prompt.

### Claim 30 (depends on 21)

The method of claim 21, wherein the generated video includes multi-object transformations based on the prompt, with cross-frame attention mechanisms ensuring that each object retains position and appearance coherence across frames, aligning with the specified attributes in the prompt.
