# US 2025/0378591 A1 · Generation of latent representations of images using a machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0378591 A1
- Document ID: US20250378591A1
- Status: Published application
- Kind: A1
- Application number: 18/738,754
- Filed: 2024-06-10
- Published: 2025-12-11
- Earliest claimed: 2024-06-10
- Assignees: Lemon Inc.
- Applicants: Lemon Inc.
- Inventors: Qihang Yu; Mark Weber; Xueqing Deng; Xiaohui Shen; Liang-Chieh (Jay) Chen
- Web page: https://patentgrep.com/patent/US20250378591A1

## CPC

- G06T 11/00: Two-dimensional [2D] image generation
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure describes techniques for generating latent representations of images using a machine learning model. An image is split and flattened into a series of patches. The series of patches is concatenated with a sequence of latent tokens. The concatenated patches and latent tokens are input into an encoder of the machine learning model. A one-dimensional (1D) latent representation of the image is generated by the encoder. Vector quantization is performed on the 1D latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens. The image is reconstructed based on the quantized latent tokens by a decoder of the machine learning model.

## Claims

### Claim 1

A method of generating latent representations of images using a machine learning model, comprising:
splitting and flattening an image into a series of patches;
concatenating the series of patches with a sequence of latent tokens;
inputting the concatenated patches and latent tokens into an encoder of the machine learning model;
generating a one-dimensional (1D) latent representation of the image by the encoder;
performing vector quantization on the 1D latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens; and
reconstructing the image based on the quantized latent tokens by a decoder of the machine learning model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the generating a one-dimensional (1D) latent representation of the image by the encoder further comprises:
decoupling a size of the 1D latent representation from a resolution of the image; and
generating the 1D latent representation comprising a 1D sequence, wherein the 1D sequence has a same size as the sequence of latent tokens.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
incorporating a sequence of mask tokens into the quantized latent tokens; and
inputting the incorporated mask tokens and quantized latent tokens into the decoder.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
generating the sequence of mask tokens by replicating a single mask token.

### Claim 5 (depends on 1)

The method of claim 1, wherein the encoder comprises a Vision Transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
training the machine learning model using a two-stage training paradigm.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
performing a first stage of training on the machine learning model using proxy codes to bypass intricate loss functions and generative adversarial network (GAN) architectures, wherein the proxy codes comprise discrete codes generated by an off-the-shelf Vector Quantized GAN (VQGAN) model.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
performing a second stage of training on the machine learning model to fine-tune the decoder, and wherein the second stage of training comprises only training the decoder towards pixel space with VQGAN losses while keeping the encoder and the vector quantizer frozen.

### Claim 9

A system of generating latent representations of images using a machine learning model, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and comprising computer-readable instructions that upon execution by the at least one processor cause the at least one processor to perform operations comprising:
splitting and flattening an image into a series of patches;
concatenating the series of patches with a sequence of latent tokens;
inputting the concatenated patches and latent tokens into an encoder of the machine learning model;
generating a one-dimensional (1D) latent representation of the image by the encoder;
performing vector quantization on the 1D latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens; and
reconstructing the image based on the quantized latent tokens by a decoder of the machine learning model.

### Claim 10 (depends on 9)

The system of claim 9, wherein the generating a one-dimensional (1D) latent representation of the image by the encoder further comprises:
decoupling a size of the 1D latent representation from a resolution of the image; and
generating the 1D latent representation comprising a 1D sequence, wherein the 1D sequence has a same size as the sequence of latent tokens.

### Claim 11 (depends on 9)

The system of claim 9, the operations further comprising:
generating a sequence of mask tokens by replicating a single mask token;
incorporating the sequence of mask tokens into the quantized latent tokens; and
inputting the incorporated mask tokens and quantized latent tokens into the decoder.

### Claim 12 (depends on 9)

The system of claim 9, wherein the encoder comprises a Vision Transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

### Claim 13 (depends on 9)

The system of claim 9, the operations further comprising:
training the machine learning model using a two-stage training paradigm.

### Claim 14 (depends on 13)

The system of claim 13, the operations further comprising:
performing a first stage of training on the machine learning model using proxy codes to bypass intricate loss functions and generative adversarial network (GAN) architectures, wherein the proxy codes comprise discrete codes generated by an off-the-shelf Vector Quantized GAN (VQGAN) model; and
performing a second stage of training on the machine learning model to fine-tune the decoder, and wherein the second stage of training comprises only training the decoder towards pixel space with VQGAN losses while keeping the encoder and the vector quantizer frozen.

### Claim 15

A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:
splitting and flattening an image into a series of patches;
concatenating the series of patches with a sequence of latent tokens;
inputting the concatenated patches and latent tokens into an encoder of the machine learning model;
generating a one-dimensional (1D) latent representation of the image by the encoder;
performing vector quantization on the 1D latent representation of the image by a vector quantizer of the machine learning model to generate quantized latent tokens; and
reconstructing the image based on the quantized latent tokens by a decoder of the machine learning model.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the generating a one-dimensional (1D) latent representation of the image by the encoder further comprises:
decoupling a size of the 1D latent representation from a resolution of the image; and
generating the 1D latent representation comprising a 1D sequence, wherein the 1D sequence has a same size as the sequence of latent tokens.

### Claim 17 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, the operations further comprising:
generating a sequence of mask tokens by replicating a single mask token;
incorporating the sequence of mask tokens into the quantized latent tokens; and
inputting the incorporated mask tokens and quantized latent tokens into the decoder.

### Claim 18 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein the encoder comprises a Vision Transformer (ViT) encoder, and wherein the decoder comprises a ViT decoder.

### Claim 19 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, the operations further comprising:
training the machine learning model using a two-stage training paradigm.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium of claim 19, the operations further comprising:
performing a first stage of training on the machine learning model using proxy codes to bypass intricate loss functions and generative adversarial network (GAN) architectures, wherein the proxy codes comprise discrete codes generated by an off-the-shelf Vector Quantized GAN (VQGAN) model; and
performing a second stage of training on the machine learning model to fine-tune the decoder, and wherein the second stage of training comprises only training the decoder towards pixel space with VQGAN losses while keeping the encoder and the vector quantizer frozen.
