# US 11,908,180 B1 · Generating videos using sequences of generative neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,180 B1 - Document ID: US11908180B1 - Status: Granted - Kind: B1 - Application number: 18/126,281 - Filed: 2023-03-24 - Granted: 2024-02-20 - Earliest claimed: 2023-03-24 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Jonathan Ho; William Chan; Chitwan Saharia; Jay Ha Whang; Tim Salimans - Web page: https://patentgrep.com/patent/US11908180B1 ## CPC - G06V 10/82: using neural networks - G06N 3/045: Combinations of networks - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium. In one aspect, a method includes receiving a text prompt describing a scene; processing the text prompt using a text encoder neural network to generate a contextual embedding of the text prompt; and processing the contextual embedding using a sequence of generative neural networks to generate a final video depicting the scene. ## Claims ### Claim 1 A method performed by one or more computers, the method comprising: receiving a text prompt describing a scene; processing the text prompt using a text encoder neural network to generate a contextual embedding of the text prompt; and processing the contextual embedding using a sequence of generative neural networks to generate a final video depicting the scene, wherein the sequence of generative neural networks comprises: an initial generative neural network configured to: receive the contextual embedding; and process the contextual embedding to generate, as output, an initial output video having: (i) an initial spatial resolution, and (ii) an initial temporal resolution; and one or more subsequent generative neural networks each configured to: receive a respective input comprising an input video generated as output by a preceding generative neural network in the sequence; and process the respective input to generate, as output, a respective output video having at least one of: (i) a higher spatial resolution, or (ii) a higher temporal resolution, than the input video, wherein the generative neural networks have been jointly trained on training data comprising a plurality of training examples that each include: (i) respective input text describing a respective scene, and (ii) a corresponding target video depicting the respective scene, wherein the training examples include image-based training examples, wherein the respective target video of each image-based training example comprises a respective plurality of individual images each depicting the respective scene described by the corresponding input text, and wherein jointly training the generative neural networks on the image-based training examples comprised masking out any temporal self-attention and temporal convolution implemented by the generative neural networks. ### Claim 2 (depends on 1) The method of claim 1, wherein the respective input of each subsequent generative neural network further comprises the contextual embedding. ### Claim 3 (depends on 1) The method of claim 1, wherein: the one or more subsequent generative neural networks comprise a plurality of subsequent generative neural networks, and the respective output video of one or more of the plurality of subsequent generative neural networks has one of: (i) a higher spatial resolution, or (ii) a higher temporal resolution, than the input video. ### Claim 4 (depends on 3) The method of claim 3, wherein the respective output video of each of the plurality of subsequent generative neural networks has one of: (i) a higher spatial resolution, or (ii) a higher temporal resolution, than the input video. ### Claim 5 (depends on 4) The method of claim 4, wherein the output videos of the plurality of subsequent generative neural networks alternate between having: (i) a higher spatial resolution, and (ii) a higher temporal resolution, than the input videos. ### Claim 6 (depends on 1) The method of claim 1, wherein video frames of each output video are generated simultaneously by their respective generative neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the text encoder neural network is pre-trained and was held frozen during the joint training of the generative neural networks. ### Claim 8 (depends on 1) The method of claim 1, wherein: the initial generative neural network implements spatial self-attention and temporal self-attention, and each subsequent generative neural network implements spatial convolution and temporal convolution. ### Claim 9 (depends on 8) The method of claim 8, wherein: the initial generative neural network further implements spatial convolution, and each subsequent generative neural network that is not a final generative neural network in the sequence further implements spatial self-attention. ### Claim 10 (depends on 1) The method of claim 1, wherein each generative neural network in the sequence is a diffusion-based generative neural network. ### Claim 11 (depends on 10) The method of claim 10, wherein the diffusion-based generative neural networks are trained using classifier-free guidance. ### Claim 12 (depends on 10) The method of claim 10, wherein each diffusion-based generative neural network uses a v-prediction parametrization to generate the respective output video. ### Claim 13 (depends on 12) The method of claim 12, wherein each diffusion-based generative neural network further uses progressive distillation to generate the respective output video. ### Claim 14 (depends on 1) The method of claim 1, wherein each subsequent generative neural network applies noise conditioning augmentation on the input video. ### Claim 15 (depends on 1) The method of claim 1, wherein the final video is the respective output video of a final generative neural network in the sequence. ### Claim 16 (depends on 1) The method of claim 1, wherein the initial spatial resolution of the initial output video corresponds to an initial per frame pixel resolution, with higher spatial resolutions corresponding to higher per frame pixel resolutions. ### Claim 17 (depends on 1) The method of claim 1, wherein the initial temporal resolution of the initial output video corresponds to an initial framerate, with higher temporal resolutions corresponding to higher framerates. ### Claim 18 A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving a text prompt describing a scene; processing the text prompt using a text encoder neural network to generate a contextual embedding of the text prompt; and processing the contextual embedding using a sequence of generative neural networks to generate a final video depicting the scene, wherein the sequence of generative neural networks comprises: an initial generative neural network configured to: receive the contextual embedding; and process the contextual embedding to generate, as output, an initial output video having: (i) an initial spatial resolution, and (ii) an initial temporal resolution; and one or more subsequent generative neural networks each configured to: receive a respective input comprising an input video generated as output by a preceding generative neural network in the sequence; and process the respective input to generate, as output, a respective output video having at least one of: (i) a higher spatial resolution, or (ii) a higher temporal resolution, than the input video, wherein the generative neural networks have been jointly trained on training data comprising a plurality of training examples that each include: (i) respective input text describing a respective scene, and (ii) a corresponding target video depicting the respective scene, wherein the training examples include image-based training examples, wherein the respective target video of each image-based training example comprises a respective plurality of individual images each depicting the respective scene described by the corresponding input text, and wherein jointly training the generative neural networks on the image-based training examples comprised masking out any temporal self-attention and temporal convolution implemented by the generative neural networks. ### Claim 19 One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving a text prompt describing a scene; processing the text prompt using a text encoder neural network to generate a contextual embedding of the text prompt; and processing the contextual embedding using a sequence of generative neural networks to generate a final video depicting the scene, wherein the sequence of generative neural networks comprises: an initial generative neural network configured to: receive the contextual embedding; and process the contextual embedding to generate, as output, an initial output video having: (i) an initial spatial resolution, and (ii) an initial temporal resolution; and one or more subsequent generative neural networks each configured to: receive a respective input comprising an input video generated as output by a preceding generative neural network in the sequence; and process the respective input to generate, as output, a respective output video having at least one of: (i) a higher spatial resolution, or (ii) a higher temporal resolution, than the input video, wherein the generative neural networks have been jointly trained on training data comprising a plurality of training examples that each include: (i) respective input text describing a respective scene, and (ii) a corresponding target video depicting the respective scene, wherein the training examples include image-based training examples, wherein the respective target video of each image-based training example comprises a respective plurality of individual images each depicting the respective scene described by the corresponding input text, and wherein jointly training the generative neural networks on the image-based training examples comprised masking out any temporal self-attention and temporal convolution implemented by the generative neural networks. ### Claim 20 (depends on 1) The method of claim 1, wherein, for each image-based training example, the respective plurality of individual images each depict a variation of the respective scene described by the corresponding input text.