# US 2025/0378682 A1 · Minimalist multi-modal approach to few-shot class-incremental learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0378682 A1
- Document ID: US20250378682A1
- Status: Published application
- Kind: A1
- Application number: 18/736,815
- Filed: 2024-06-07
- Published: 2025-12-11
- Earliest claimed: 2024-06-07
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Thang DOAN; Sima BEHPOUR; Xin LI; Wenbin HE; Liang GOU; Liu REN
- Web page: https://patentgrep.com/patent/US20250378682A1

## CPC

- G06V 10/82: using neural networks
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for Few-Shot Class-Incremental Learning (FSCIL) that utilizes a combination of Session Specific Prompts (SSP) and hyperbolic distance metrics to enhance session-wise learning and representation of image-text pairings across differing classes. The methods and systems include a base training session where both text and image features are projected into hyperbolic space for accurate class pairing using a cross-entropy loss function. Subsequent incremental sessions incorporate previously learned SSPs to retain and augment the separability of classes while minimizing the trainable parameters. This enhances performance in image-text classification tasks by leveraging a minimalistic approach, achieving higher accuracy with fewer trainable parameters compared to traditional models.

## Claims

### Claim 1

A computer-implemented method for training a machine-learning system with few-shot class-incremental learning, the method comprising:
providing a Contrastive Language-Image Pretraining (CLIP) model having (i) an image encoder configured to generate image embeddings associated with input images, (ii) a text encoder configured to generate text embeddings associated with text input, and (iii) a comparison module configured to compare similarities amongst the image embeddings and the text embeddings;
during a base training session:
(i) adjusting vision prompt weights within the image encoder and text prompt weights within the text encoder,
(ii) executing the CLIP model on the input images and the text input with the adjusted vision prompt weights and text prompt weights to generate base session image embeddings and base session text embeddings,
(iii) pairing the base session image embeddings and the base session text embeddings using a base session cross-entropy loss function, and
(iv) repeating the base training session to minimize the base session cross-entropy loss function;
during incremental sessions after the base training session:
(i) freezing the vision prompt weights while leaving the text prompt weights unfrozen,
(ii) executing the CLIP model with the frozen vision prompt weights and the unfrozen text prompt weights to generate incremental session image embeddings and incremental session text embeddings,
(iii) pairing the incremental session image embeddings and the incremental session text embeddings using an incremental session cross-entropy loss function with session-specific prompt weights associated with each respective incremental session, and
(iv) repeating the incremental sessions to minimize the incremental session cross-entropy loss function utilizing the session-specific prompt weights; and
outputting a trained machine-learning system based on the base training session and the incremental sessions.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
during the base training session, projecting the base session image embeddings and the base session text embeddings into a hyperbolic space;
wherein the pairing of the base session image embeddings and the base session text embeddings is based on the hyperbolic space.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
determining a hyperbolic distance between the base session image embeddings and the base session text embeddings;
wherein the pairing of the base session image embeddings and the base session text embeddings is based on the hyperbolic distance.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the image encoder includes transformer layers,
the text encoder includes transformer layers,
the step of adjusting includes:
intersplicing the vision prompts between the transformer layers of the image encoder, and
intersplicing the text prompt weights between the transformer layers of the text encoder.

### Claim 5 (depends on 1)

The method of claim 1, wherein the vision prompts comprise vectorized visual data, and the text prompt weights comprise vectorized textual data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
executing a regularization configured to maintain pre-learned knowledge by imposing constraints on deviations from the vision prompt weights and the text prompt weights.

### Claim 7 (depends on 6)

The method of claim 6, wherein the executing of the regularization includes calculating a loss term for discrepancies between adjusted image prompt weights and the frozen image prompt weights.

### Claim 8 (depends on 1)

The method of claim 1, wherein the session-specific prompt weights in the incremental sessions are generated based on the text embeddings and are differentiated from the text prompt weights used in previous sessions.

### Claim 9 (depends on 8)

The method of claim 8, wherein the session-specific prompt weights are incorporated into the incremental session cross-entropy loss function.

### Claim 10 (depends on 9)

The method of claim 9, further comprising saving the session-specific prompt weights as non-differentiable data and reusing the session-specific prompt weights in subsequent incremental sessions as parameters of the incremental session cross-entropy loss function.

### Claim 11 (depends on 10)

The method of claim 10, wherein saving the session-specific prompt weights involves storing them in a buffer configured to retain session-specific information across multiple incremental sessions.

### Claim 12

A system for training a machine-learning models with few-shot class-incremental learning, the system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to perform the following steps:
providing a Contrastive Language-Image Pretraining (CLIP) model having (i) an image encoder configured to generate image embeddings associated with input images, (ii) a text encoder configured to generate text embeddings associated with text input, and (iii) a comparison module configured to compare similarities amongst the image embeddings and the text embeddings;
during a base training session:
(i) adjusting vision prompt weights within the image encoder and text prompt weights within the text encoder,
(ii) executing the CLIP model on the input images and the text input with the adjusted vision prompt weights and text prompt weights to generate base session image embeddings and base session text embeddings,
(iii) pairing the base session image embeddings and the base session text embeddings using a base session cross-entropy loss function, and
(iv) repeating the base training session to minimize the base session cross-entropy loss function;
during incremental sessions after the base training session:
(i) freezing the vision prompt weights while leaving the text prompt weights unfrozen,
(ii) executing the CLIP model with the frozen vision prompt weights and the unfrozen text prompt weights to generate incremental session image embeddings and incremental session text embeddings,
(iii) pairing the incremental session image embeddings and the incremental session text embeddings using an incremental session cross-entropy loss function with session-specific prompt weights associated with each respective incremental session, and
(iv) repeating the incremental sessions to minimize the incremental session cross-entropy loss function utilizing the session-specific prompt weights; and
outputting a trained machine-learning system based on the base training session and the incremental sessions.

### Claim 13 (depends on 12)

The system of claim 12, wherein the instructions, when executed by the processor, further cause the processor to perform the following step:
during the base training session, projecting the base session image embeddings and the base session text embeddings into a hyperbolic space;
wherein the pairing of the base session image embeddings and the base session text embeddings is based on the hyperbolic space.

### Claim 14 (depends on 13)

The system of claim 13, wherein the instructions, when executed by the processor, further cause the processor to perform the following step:
determining a hyperbolic distance between the base session image embeddings and the base session text embeddings;
wherein the pairing of the base session image embeddings and the base session text embeddings is based on the hyperbolic distance.

### Claim 15 (depends on 12)

The system of claim 12, wherein:
the image encoder includes transformer layers,
the text encoder includes transformer layers,
the step of adjusting includes:
intersplicing the vision prompts between the transformer layers of the image encoder, and
intersplicing the text prompt weights between the transformer layers of the text encoder.

### Claim 16 (depends on 12)

The system of claim 12, wherein the vision prompts comprise vectorized visual data, and the text prompt weights comprise vectorized textual data.

### Claim 17 (depends on 12)

The system of claim 12, wherein the instructions, when executed by the processor, further cause the processor to perform the following step:
executing a regularization configured to maintain pre-learned knowledge by imposing constraints on deviations from the vision prompt weights and the text prompt weights.

### Claim 18 (depends on 17)

The system of claim 17, wherein the executing of the regularization includes calculating a loss term for discrepancies between adjusted image prompt weights and the frozen image prompt weights.

### Claim 19 (depends on 12)

The system of claim 12, wherein the session-specific prompt weights in the incremental sessions are generated based on the text embeddings and are differentiated from the text prompt weights used in previous sessions.

### Claim 20

A non-transitory computer-readable medium having instructions stored thereon, which when executed by a processor, causes a machine-learning system to perform a method for training with few-shot class-incremental learning, the method comprising:
providing a Contrastive Language-Image Pretraining (CLIP) model having (i) an image encoder configured to generate image embeddings associated with input images, (ii) a text encoder configured to generate text embeddings associated with text input, and (iii) a comparison module configured to compare similarities amongst the image embeddings and the text embeddings;
during a base training session:
(i) adjusting vision prompt weights within the image encoder and text prompt weights within the text encoder,
(ii) executing the CLIP model on the input images and the text input with the adjusted vision prompt weights and text prompt weights to generate base session image embeddings and base session text embeddings,
(iii) pairing the base session image embeddings and the base session text embeddings using a base session cross-entropy loss function, and
(iv) repeating the base training session to minimize the base session cross-entropy loss function;
during incremental sessions after the base training session:
(i) freezing the vision prompt weights while leaving the text prompt weights unfrozen,
(ii) executing the CLIP model with the frozen vision prompt weights and the unfrozen text prompt weights to generate incremental session image embeddings and incremental session text embeddings,
(iii) pairing the incremental session image embeddings and the incremental session text embeddings using an incremental session cross-entropy loss function with session-specific prompt weights associated with each respective incremental session, and
(iv) repeating the incremental sessions to minimize the incremental session cross-entropy loss function utilizing the session-specific prompt weights; and
outputting a trained machine-learning system based on the base training session and the incremental sessions.
