# US 12,192,547 B2 · High-resolution video generation using image diffusion models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,547 B2 - Document ID: US12192547B2 - Status: Granted - Kind: B2 - Application number: 18/181,729 - Filed: 2023-03-10 - Published: 2024-05-23 - Granted: 2025-01-07 - Earliest claimed: 2022-11-16 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Karsten Julian Kreis; Robin Rombach; Andreas Blattmann; Seung Wook Kim; Huan Ling; Sanja Fidler; Tim Dockhorn - Web page: https://patentgrep.com/patent/US12192547B2 ## CPC - H04N 21/234363: {by altering the spatial resolution, e.g. for clients with a lower screen resolution} - G06N 3/045: Combinations of networks - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI] - G06V 10/82: using neural networks - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - H04N 7/0117: {involving conversion of the spatial resolution of the incoming video signal (for graphics images G09G2340/0407)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In various examples, systems and methods are disclosed relating to aligning images into frames of a first video using at least one first temporal attention layer of a neural network model. The first video has a first spatial resolution. A second video having a second spatial resolution is generated by up-sampling the first video using at least one second temporal attention layer of an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution. ## Claims ### Claim 1 A processor, comprising: one or more circuits to: align a plurality of images into frames of a first video using a neural network model comprising a latent diffusion model (LDM), wherein the first video has a first spatial resolution, the LDM comprises: an encoder to map an input from an image space to a latent space; and a decoder to map latent encoding from the latent space to the image space; and generate a second video having a second spatial resolution by up-sampling the first video using an up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution, wherein the decoder is updated according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space. ### Claim 2 (depends on 1) The processor of claim 1, wherein the neural network model is modified from an image diffusion model by adding at least one first temporal attention layer into the image diffusion model. ### Claim 3 (depends on 1) The processor of claim 1, wherein the up-sampler neural network model comprises a first diffusion model for video generation, and the first diffusion model is modified from a second diffusion model for image generation by adding at least one second temporal attention layer into the first diffusion model for image generation. ### Claim 4 (depends on 1) The processor of claim 1, wherein the plurality of images are consecutive frames of the first video. ### Claim 5 (depends on 1) The processor of claim 1, wherein: the neural network model comprises a first diffusion model and a second diffusion model; the first diffusion model is to generate a third video; and the second diffusion model is to generate the first video by generating at least one frame between two consecutive frames of the third video. ### Claim 6 (depends on 1) The processor of claim 1, wherein the neural network model is to: generate a third video; and generate the first video by generating at least one frame between two consecutive frames of the third video according to relative time step embedding. ### Claim 7 (depends on 1) The processor of claim 1, wherein the first video is generated according to at least one of: one or more text prompts; one or more bounding boxes; or one or more conditioning signals. ### Claim 8 A processor, comprising: one or more circuits to: update a neural network model to align a plurality of images into frames of a first video by updating at least one layer of the neural network model, wherein the first video has a first spatial resolution, wherein the neural network model comprises a Latent Diffusion Model (LDM) comprising: an encoder to map an input from an image space to a latent space; and a decoder to map latent encoding from the latent space to the image space, wherein to update the neural network model, the one or more processing units are to update the decoder according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space; and update an up-sampler neural network model to generate a second video having a second spatial resolution via up-sampling the first video by updating at least one layer of the up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution. ### Claim 9 (depends on 8) The processor of claim 8, wherein the one or more circuits are to: insert the at least one neural network layer into a denoising neural network of an image diffusion model; and update the at least one neural network layer to align the plurality of images into the frames of the first video. ### Claim 10 (depends on 8) The processor of claim 8, wherein the one or more circuits are to: update at least one other layer of the neural network model using a plurality of sample images; and update the at least one layer using at least one sample sequence of images, while maintaining parameters of the at least one other layer unchanged. ### Claim 11 (depends on 8) The processor of claim 8, wherein the plurality of images are consecutive frames of the first video. ### Claim 12 (depends on 8) The processor of claim 8, wherein: the neural network model comprises a first diffusion model and a second diffusion model; the first diffusion model is updated to generate a third video; and the second diffusion model is updated to generate the first video by generating at least one frame between two consecutive frames of the third video. ### Claim 13 (depends on 8) The processor of claim 8, wherein: the neural network model is updated to generate a third video; and the neural network model is updated to generate the first video by generating at least one frame between two consecutive frames of the third video according to relative time step embedding. ### Claim 14 (depends on 8) The processor of claim 8, wherein the one or more circuits are to update the neural network model to generate the first video according to at least one of: one or more text prompts; one or more bounding boxes; or one or more conditioning signals. ### Claim 15 (depends on 8) The processor of claim 8, wherein the one or more circuits are to update the neural network model to generate the first video according to modified conditioning signals for classifier-free guidance. ### Claim 16 A method, comprising: updating a neural network model to align a plurality of images into frames of a first video by updating at least one first temporal attention layer of the neural network model, wherein the first video has a first spatial resolution, wherein the neural network model comprises a Latent Diffusion Model (LDM) comprising: an encoder to map an input from an image space to a latent space; and a decoder to map latent encoding from the latent space to the image space, wherein updating the neural network model comprises updating the decoder according to one or more temporal incoherencies in mapping the latent encoding from the latent space to the image space; and updating an up-sampler neural network model to generate a second video having a second spatial resolution via up-sampling the first video, by updating at least one second temporal attention layer of the up-sampler neural network model, wherein the second spatial resolution is higher than the first spatial resolution.