US 2026/0073692 A1 · POSITIONAL EMBEDDING AND TRAINING TECHNIQUES FOR A DIFFUSION MODEL

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates spatial-temporal positional encodings. For example, the disclosed systems generate a noised token from adding noise to an embedding of a frame of a video. Moreover, the disclosed systems generate a spatial embedding for a token using a centered two-dimensional coordinate map. Further, the disclosed systems generate temporal embeddings for the token from a timestamp of the token in the video. Further, the disclosed systems generate a denoised token by removing noise from the noised token according to spatial-temporal positional encodings that include the spatial embedding and the temporal embedding via a diffusion model. Additionally, the disclosed systems modify parameters of the diffusion model based on a comparison of the denoised token and the token.

Claims20

Dependent claims

A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

generating, from a video, a noised token based on adding noise to an embedding of a frame of the video;
generating, utilizing a centered two-dimensional coordinate map, a spatial embedding for the noised token of a sequence of tokens to index a location of the noised token within the frame from which the noised token was generated;
generating a temporal embedding for the noised token of the sequence of tokens from a timestamp;
generating, utilizing a diffusion model, a denoised token by denoising the noised token according to spatial-temporal positional encodings comprising the spatial embedding and the temporal embedding;
and
modifying parameters of the diffusion model based on the denoised token.

3 more elements

A computer-implemented method comprising:

generating, from a video, an embedding of a frame of the video;
generating a noised token from the embedding by adding noise to the embedding and further tokenizing the embedding;
generating, utilizing a centered two-dimensional coordinate map, a spatial embedding for the noised token to index a location of the noised token within the frame from which the noised token was generated;
generating a temporal embedding for the noised token from a timestamp of the noised token in the video;
generating, utilizing a diffusion model, a denoised token by removing noise from the noi sed token according to a block specific embedding for the noised token and spatial-temporal positional encodings comprising the spatial embedding and the temporal embedding;
and
modifying parameters of the diffusion model based on the denoised token.

4 more elements

Dependencies are inferred from the claim text.

Positional embedding and training techniques for a diffusion model

Life of this patent

Positional embedding and training techniques for a diffusion model

Life of this patent

Abstract

Claims20