Video tokenization using channel-split quantization and mamba-based tokenizer models
AssigneeNVIDIA Corporation
InventorsDawit Mureja ARGAW, Fitsum REDA, Ming-Yu LIU
Life of this patent
Estimating the term…
Abstract
The disclosed method for tokenizing video frames includes receiving one or more video frames from one or more I/O devices, generating, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules, and generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.
Claims20
A computer-implemented method for tokenizing video frames, the method comprising:
- receiving one or more video frames from one or more I/O devices;
- generating, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules;
- and
- generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.
One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
- receiving one or more video frames from one or more I/O devices;
- generating, using an encoder, one or more latent embeddings, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules based on the one or more video frames;
- and
- generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.
A system, comprising:
- one or more memories storing instructions, and
- one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
- receive one or more video frames from one or more I/O devices;
- generate, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules, and
- generate, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.
Dependencies are inferred from the claim text.