# US 2026/0087351 A1 · Video tokenization using channel-split quantization and mamba-based tokenizer models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087351 A1
- Document ID: US20260087351A1
- Status: Published application
- Kind: A1
- Application number: 19/301,848
- Filed: 2025-08-15
- Published: 2026-03-26
- Earliest claimed: 2024-09-24
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Dawit Mureja ARGAW; Fitsum REDA; Ming-Yu LIU
- Web page: https://patentgrep.com/patent/US20260087351A1

## CPC

- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4046: using neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The disclosed method for tokenizing video frames includes receiving one or more video frames from one or more I/O devices, generating, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules, and generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.

## Claims

### Claim 1

A computer-implemented method for tokenizing video frames, the method comprising:
receiving one or more video frames from one or more I/O devices;
generating, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules; and
generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the one or more latent embeddings comprises:
generating, using a first patchify module included in the one or more patchify modules based on the one or more video frames, one or more first patched tokens;
generating, using a first spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, one or more processed patched tokens based on the one or more first patched tokens;
generating, using a second patchify module included in the one or more patchify modules, one or more second patched tokens based on the one or more processed patched tokens; and
generating, using a first token pooling module included in the one or more token pooling modules, one or more first pooled tokens based on the one or more second patched tokens and the one or more processed patched tokens.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising:
generating, using a second spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, one or more processed pooled tokens based on the one or more first pooled tokens;
generating, using a third patchify module included in the one or more patchify modules, one or more third patched tokens based on the one or more processed pooled tokens;
generating, using a second token pooling module included in the one or more token pooling modules, one or more second pooled tokens based on the one or more third patched tokens and the one or more processed pooled tokens; and
generating, using a third spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, the one or more latent embeddings based on the one or more second pooled tokens.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating, using a decoder, one or more reconstructed video frames based on the one or more quantized latent embeddings, wherein the decoder comprises one or more temporal-spatial Mamba modules, one or more topixel modules, and one or more token interpolation modules.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein at least a first token interpolation module included in the one or more token interpolation modules performs:
a nearest-neighbor interpolation in one or more temporal and spatial dimensions to match a resolution of one or more processed tokens to generate one or more upsampled tokens; and
elementwise adding of the one or more upsampled tokens and the one or more processed tokens to generate one or more interpolated tokens.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein generating the one or more reconstructed video frames comprises:
generating, using a first temporal-spatial Mamba module included in the one or more temporal-spatial Mamba modules, one or more first processed tokens based on the one or more quantized latent embeddings;
generating, using a first embedding layer included in a first topixel module included in the one or more topixel modules, one or more first projected tokens based on the one or more first processed tokens;
generating, using a first pixelshuffle layer included in the first topixel module included in the one or more topixel modules, one or more first grid-like tokens based on the first projected tokens;
generating, using a first token interpolation module included in the one or more token interpolation modules, one or more first interpolated tokens based on the one or more first grid-like tokens and the one or more first processed tokens; and
generating, using a second temporal-spatial Mamba module included in the one or more temporal-spatial Mamba modules, one or more second processed tokens based on the one or more first interpolated tokens.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
generating, using a second embedding layer included in a second topixel module included in the one or more topixel modules, one or more second projected tokens based on the one or more second processed tokens;
generating, using a second pixelshuffle layer included in the second topixel module included in the one or more topixel modules, one or more second grid-like tokens based on the second projected tokens;
generating, using a second token interpolation module included in the one or more token interpolation modules, one or more second interpolated tokens based on the one or more second grid-like tokens and the one or more second processed tokens;
generating, using a third temporal-spatial Mamba module included in the one or more temporal-spatial Mamba modules, one or more third processed tokens based on the one or more second interpolated tokens; and
generating, using a third topixel module included in the one or more third topixel modules, the one or more reconstructed video frames based on the one or more third processed tokens.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein a first patchify module included in the one or more patchify modules comprises at least one of a reshape layer or an embedding layer.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein a 3D convolutional layer included in the first patchify module applies a kernel of fixed size across one or more non-overlapping windows of one or more processed pooled tokens to generate one or more patched tokens.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein a first token pooling module included in the one or more token pooling modules performs
downsampling of one or more processed patched tokens using 3D average pooling with a fixed kernel size to generate one or more downsampled tokens; and
adding the one or more downsampled tokens and one or more patched tokens to generate one or more pooled tokens.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein a first spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules comprises a first number of one or more stacked spatial Mamba layers followed by the first number of one or more temporal Mamba layers.

### Claim 12

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving one or more video frames from one or more I/O devices;
generating, using an encoder, one or more latent embeddings, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules based on the one or more video frames; and
generating, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein generating the one or more latent embeddings comprises:
generating, using a first patchify module included in the one or more patchify modules, one or more first patched tokens based on the one or more video frames;
generating, using a first spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, one or more processed patched tokens based on the one or more first patched tokens;
generating, using a second patchify module included in the one or more patchify modules, one or more second patched tokens based on the one or more processed patched tokens; and
generating, using a first token pooling module included in the one or more token pooling modules, one or more first pooled tokens based on the one or more second patched tokens and the one or more processed patched tokens.

### Claim 14 (depends on 13)

The one or more non-transitory computer-readable media of claim 13, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the steps of:
generating, using a second spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, one or more processed pooled tokens based on the one or more first pooled tokens;
generating, using a third patchify module included in the one or more patchify modules, one or more third patched tokens based on the one or more processed pooled tokens;
generating, using a second token pooling module included in the one or more token pooling modules, one or more second pooled tokens based on the one or more third patched tokens and the one or more processed pooled tokens; and
generating, using a third spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules, the one or more latent embeddings based on the one or more second pooled tokens.

### Claim 15 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein a first patchify module included in the one or more patchify modules comprises at least one of a reshape layer or an embedding layer.

### Claim 16 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein a first token pooling module included in the one or more token pooling modules performs:
downsampling of one or more processed patched tokens using 3D average pooling with a fixed kernel size to generate one or more downsampled tokens; and
adding the one or more downsampled tokens and one or more patched tokens to generate one or more pooled tokens.

### Claim 17 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein a first spatial-temporal Mamba module included in the one or more spatial-temporal Mamba modules comprises a first number of one or more temporal Mamba layers followed by the first number of one or more spatial Mamba layers.

### Claim 18 (depends on 12)

The one or more non-transitory computer-readable media of claim 12, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of:
generating, using a decoder, one or more reconstructed video frames based on the one or more quantized latent embeddings, wherein the decoder comprises one or more temporal-spatial Mamba modules, one or more topixel modules, and one or more token interpolation modules.

### Claim 19 (depends on 18)

The one or more non-transitory computer-readable media of claim 18, wherein at least a first token interpolation module included in the one or more token interpolation modules performs:
a nearest-neighbor interpolation in one or more temporal and spatial dimensions to match a resolution of one or more processed tokens to generate one or more upsampled tokens; and
elementwise adding of the one or more upsampled tokens and the one or more processed tokens to generate one or more interpolated tokens.

### Claim 20

A system, comprising:
one or more memories storing instructions, and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
receive one or more video frames from one or more I/O devices;
generate, using an encoder, one or more latent embeddings based on the one or more video frames, wherein the encoder comprises one or more patchify modules, one or more spatial-temporal Mamba modules, and one or more token pooling modules, and
generate, using a quantizer, one or more quantized latent embeddings based on the one or more latent embeddings.
