Temporally consistent and semantics guided text-based video editing generative artificial intelligence (AI) model with improved initialization
AssigneeQUALCOMM Incorporated
InventorsHyojin Park, Debasmit Das, Munawar Hayat, Fatih Murat Porikli
Life of this patent
Estimating the term…
Abstract
A processor-implemented method performed for text-based video editing includes receiving a video input and a text prompt. The video input includes a sequence of video frames. Features of the video input are extracted to generate a latent representation of the video input. Noise is injected to the latent representation of the video input to generate a noise injected latent. The noise is conditioned on the video input. An artificial neural network (ANN) model processes the noise injected latent based on the text prompt to adapt the video input according to the text prompt.
Claims17
An apparatus, comprising:
- at least one memory;
- and
- at least one processor coupled to the at least one memory, the at least one processor configured to:
- receive a video input and a text prompt, the video input including a sequence of video frames;
- extract features of the video input to generate a latent representation of the video input;
- inject noise to the latent representation of the video input to generate a noise injected latent, the noise injected latent being generated recursively based on a weighted sum of noisy latent vectors, the noise being conditioned on frames of the video input;
- process, by an artificial neural network (ANN) model, the noise injected latent based on the text prompt to adapt the video input according to the text prompt using fixed parameters of the ANN model;
- and
- generate an adapted video output maintaining temporal consistency across the sequence of video frames.
A processor-implemented method performed by at least one processor, the processor-implemented method comprising:
- receiving a video input and a text prompt, the video input including a sequence of video frames;
- extracting features of the video input to generate a latent representation of the video input;
- injecting noise to the latent representation of the video input to generate a noise injected latent, the noise injected latent being generated recursively based on a weighted sum of noisy latent vectors, the noise being conditioned on frames of the video input;
- processing, by an artificial neural network (ANN) model, the noise injected latent based on the text prompt to adapt the video input according to the text prompt using fixed parameters of the ANN model;
- and
- generating an adapted video output maintaining temporal consistency across the sequence of video frames.
An apparatus, comprising:
- means for receiving a video input and a text prompt, the video input including a sequence of video frames;
- means for extracting features of the video input to generate a latent representation of the video input;
- means for injecting noise to the latent representation of the video input to generate a noise injected latent, the noise injected latent being generated recursively based on a weighted sum of noisy latent vectors, the noise being conditioned on frames of the video input;
- means for processing, by an artificial neural network (ANN) model, the noise injected latent based on the text prompt to adapt the video input according to the text prompt using fixed parameters of the ANN model;
- and
- means for generating an adapted video output maintaining temporal consistency across the sequence of video frames.
Dependencies are inferred from the claim text.