# US 2026/0087701 A1 · Video editing using diffusion models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2026/0087701 A1 - Document ID: US20260087701A1 - Status: Published application - Kind: A1 - Application number: 19/286,151 - Filed: 2025-07-30 - Published: 2026-03-26 - Earliest claimed: 2023-01-31 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yaniv Leviathan; Eyal Molad; Eliahu Moshe Horwitz; Daniel Walevski; Alexander Rav Acha; Yossi Matias; Yedid Hoshen; Yael Pritch Knaan - Web page: https://patentgrep.com/patent/US20260087701A1 ## CPC - G06T 11/60: Creating or editing images; Combining images with text - G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06T 11/00: Two-dimensional [2D] image generation - G06T 13/00: Animation - G06T 2211/441: AI-based methods, deep learning or artificial neural networks - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output video. One of the methods include: obtaining an input video; obtaining input text that includes a description of an output video; generating, based at least on applying downsampling to the input video, a degraded version of the input video; and generating the output video based on the description in the input text by updating the degraded version of the input video by using a video diffusion model across a plurality of reverse diffusion steps. ## Claims ### Claim 1 A computer-implemented method of generating an output video, the method comprising: obtaining an input video; obtaining input text that includes a description of an output video; generating, based at least on applying downsampling to the input video, a degraded version of the input video; and generating the output video based on the description in the input text by updating the degraded version of the input video, wherein the updating comprises, at each of a plurality of steps: processing, by a diffusion model, a diffusion model input comprising (i) a current intermediate representation of the output video and (ii) the input text to generate a noise output for the step; and using the noise output to de-noise the current intermediate representation of the output video to generate an updated intermediate representation of the output video for the step. ### Claim 2 (depends on 1) The method of claim 1, wherein generating the degraded version of the input video comprises: generating a downsampled version of the input video by applying downsampling to the input video; and generating the degraded version of the input video by adding Gaussian noise with a predetermined variance to the downsampled version of the input video. ### Claim 3 (depends on 1) The method of claim 1, wherein obtaining the input video comprises: receiving from a client device the input video that has a plurality of video frames. ### Claim 4 (depends on 1) The method of claim 1, wherein obtaining the input video comprises: receiving from a client device one or more input images; generating a synthetic video that has a plurality of video frames by replicating, transforming, or both each of the one or more input images; and using the synthetic video as the input video. ### Claim 5 (depends on 1) The method of claim 1, wherein the diffusion model has pre-trained parameter values, and generating the output video based on the description in the input text comprises: fine-tuning the diffusion model with respect to the input video to adjust the pre-trained parameter values of the diffusion model. ### Claim 6 (depends on 5) The method of claim 5, wherein fine-tuning the diffusion model with respect to the input video comprises: adjusting the pre-trained parameter values of the diffusion model based on optimizing a mixed fine-tuning objective function that includes a first term that evaluates a difference between (i) the input video and (ii) a reconstructed representation of the input video generated by using the diffusion model, and a second term that evaluates a difference between (i) each of one or more frames of the input video and (ii) a reconstructed representation of each of the one or more frames of the input video generated by using the diffusion model. ### Claim 7 (depends on 5) The method of any one of claim 5, wherein fine-tuning the diffusion model with respect to the input video comprises: generating a unique identifier for a subject instance depicted in the input video; and processing the unique identifier as the input text by the diffusion model during the fine-tuning. ### Claim 8 (depends on 5) The method of any one of claim 5, wherein the diffusion model comprises (i) one or more temporal attention layers that each attend over the plurality of video frames in the input video when generating a corresponding attention layer output and (ii) one or more one or more spatial attention layers that each attend over a plurality of pixels in a video frame when generating a corresponding spatial attention layer output, and wherein fine-tuning the diffusion model on the input video comprises: adjusting the pre-trained parameter values of the one or more spatial attention layers while holding the pre-trained parameter values of the one or more temporal attention layers fixed. ### Claim 9 (depends on 1) The method of any one of claim 1, wherein the output video and the input video both depict a subject instance but a motion, an appearance, or both of the subject instance are different. ### Claim 10 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: obtaining an input video; obtaining input text that includes a description of an output video; generating, based at least on applying downsampling to the input video, a degraded version of the input video; and generating the output video based on the description in the input text by updating the degraded version of the input video, wherein the updating comprises, at each of a plurality of steps: processing, by a diffusion model, a diffusion model input comprising (i) a current intermediate representation of the output video and (ii) the input text to generate a noise output for the step; and using the noise output to de-noise the current intermediate representation of the output video to generate an updated intermediate representation of the output video for the step. ### Claim 11 A computer storage medium encoded with a computer program, the program comprising instructions that are operable, when executed by data processing apparatus, to cause the data processing apparatus to perform operations comprising: obtaining an input video; obtaining input text that includes a description of an output video; generating, based at least on applying downsampling to the input video, a degraded version of the input video; and generating the output video based on the description in the input text by updating the degraded version of the input video, wherein the updating comprises, at each of a plurality of steps: processing, by a diffusion model, a diffusion model input comprising (i) a current intermediate representation of the output video and (ii) the input text to generate a noise output for the step; and using the noise output to de-noise the current intermediate representation of the output video to generate an updated intermediate representation of the output video for the step. ### Claim 12 (depends on 10) The system of claim 10, wherein: the diffusion model has pre-trained parameter values, and generating the output video based on the description in the input text comprises fine-tuning the diffusion model with respect to the input video to adjust the pre-trained parameter values of the diffusion model; and fine-tuning the diffusion model with respect to the input video comprises: adjusting the pre-trained parameter values of the diffusion model based on optimizing a mixed fine-tuning objective function that includes a first term that evaluates a difference between (i) the input video and (ii) a reconstructed representation of the input video generated by using the diffusion model, and a second term that evaluates a difference between (i) each of one or more frames of the input video and (ii) a reconstructed representation of each of the one or more frames of the input video generated by using the diffusion model. ### Claim 13 (depends on 12) The system of claim 12, wherein fine-tuning the diffusion model with respect to the input video comprises: generating a unique identifier for a subject instance depicted in the input video; and processing the unique identifier as the input text by the diffusion model during the fine-tuning. ### Claim 14 (depends on 12) The system of claim 12, wherein the diffusion model comprises (i) one or more temporal attention layers that each attend over the plurality of video frames in the input video when generating a corresponding attention layer output and (ii) one or more one or more spatial attention layers that each attend over a plurality of pixels in a video frame when generating a corresponding spatial attention layer output, and wherein fine-tuning the diffusion model on the input video comprises: adjusting the pre-trained parameter values of the one or more spatial attention layers while holding the pre-trained parameter values of the one or more temporal attention layers fixed. ### Claim 15 (depends on 11) The computer storage medium of claim 11, wherein: the diffusion model has pre-trained parameter values, and generating the output video based on the description in the input text comprises fine-tuning the diffusion model with respect to the input video to adjust the pre-trained parameter values of the diffusion model; and fine-tuning the diffusion model with respect to the input video comprises: adjusting the pre-trained parameter values of the diffusion model based on optimizing a mixed fine-tuning objective function that includes a first term that evaluates a difference between (i) the input video and (ii) a reconstructed representation of the input video generated by using the diffusion model, and a second term that evaluates a difference between (i) each of one or more frames of the input video and (ii) a reconstructed representation of each of the one or more frames of the input video generated by using the diffusion model. ### Claim 16 (depends on 15) The computer storage medium of claim 15, wherein fine-tuning the diffusion model with respect to the input video comprises: generating a unique identifier for a subject instance depicted in the input video; and processing the unique identifier as the input text by the diffusion model during the fine-tuning. ### Claim 17 (depends on 15) The computer storage medium of claim 15, wherein the diffusion model comprises (i) one or more temporal attention layers that each attend over the plurality of video frames in the input video when generating a corresponding attention layer output and (ii) one or more one or more spatial attention layers that each attend over a plurality of pixels in a video frame when generating a corresponding spatial attention layer output, and wherein fine-tuning the diffusion model on the input video comprises: adjusting the pre-trained parameter values of the one or more spatial attention layers while holding the pre-trained parameter values of the one or more temporal attention layers fixed.