# US 2026/0087369 A1 · Method and apparatus for continuous learning using asymmetric structures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087369 A1
- Document ID: US20260087369A1
- Status: Published application
- Kind: A1
- Application number: 19/255,664
- Filed: 2025-06-30
- Published: 2026-03-26
- Earliest claimed: 2024-09-24
- Assignees: INFINIQ Co., Ltd.
- Applicants: INFINIQ Co., Ltd.
- Inventors: Min Jae JUNG; Joo Hee KIM; Seung Jin OH; Seung Taek KIM; Kyung Shil KANG
- Web page: https://patentgrep.com/patent/US20260087369A1

## CPC

- G06N 3/096: Transfer learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present invention aims to minimize catastrophic forgetting that occurs during the continual learning process of a large language model (LLM), and to improve the model's performance by efficiently acquiring new knowledge. A memory of a continual learning apparatus using an asymmetric structure according to one embodiment of the present invention may store datasets used for continual learning, model parameters, a router, and adapters. At least one processor may be configured to perform continual learning using a neural network model comprising a shallow layer and a deep layer; to add a new adapter corresponding to new learning to the deep layer whenever new data is learned; and to distribute input text to at least one of the shallow layer, the deep layer, and the adapter via a router provided between the shallow layer and the deep layer.

## Claims

### Claim 1

A continual learning apparatus using an asymmetric structure, the continual learning apparatus comprising:
a memory configured to store a dataset, model parameters, a router, and adapters used for continual learning; and
at least one processor configured to communicate with the memory,
wherein the at least one processor is configured to:
perform continual learning using a neural network model comprising a shallow layer and a deep layer,
add an adapter corresponding to new learning to the deep layer whenever new data is learned,
distribute input text to at least one of the shallow layer, the deep layer, and the adapter via the router provided between the shallow layer and the deep layer,
perform regularization on the adapter to improve learning efficiency of the neural network model,
perform the regularization using an orthogonal loss function to maintain independence of information between different adapters,
perform the regularization using the orthogonal loss function by calculating orthogonality between output data of different adapters and projecting the output data onto different planes,
perform Sparse Low-rank Adaptation (SoRA) based on a gate vector provided between the adapters, and
determine a capacity of each of the adapters based on characteristics of the input text,
wherein the gate vector is composed of a vector in a rank dimension,
wherein the gate vector adjusts the rank by performing a Hadamard product with a feature vector that has passed through a predetermined layer,
wherein the orthogonal loss function is determined by Equation 1 as follows:
ℒ
=
∑
i
<
j
A
i
·
A
j
[
Equation
⁢
1
]
wherein L denotes a loss function, and Ai and Aj denote an i-th and a j-th A adapter, respectively,
wherein the Hadamard product is performed based on Equation 2 as follows:
h
=
A
·
(
g
⊙
(
B
·
x
)
)
[
Equation
⁢
2
]
wherein h represents a forward pass of each adapter, A and B represent parameters constituting each adapter, g represents the vector in the rank dimension, and x represents the feature vector that has passed through the predetermined layer.

### Claim 2 (depends on 1)

The continual learning apparatus using the asymmetric structure of claim 1, wherein the at least one processor is configured to:
distribute the input text based on a feature corresponding to the deep layer via the router.

### Claim 3 (depends on 1)

The continual learning apparatus using the asymmetric structure of claim 1, wherein the at least one processor is configured to:
improve parameter efficiency by using Low-Rank Adaptation (LoRA) through the adapter provided in the deep layer and using at least one portion of the neural network model.

### Claim 4 (depends on 1)

The continual learning apparatus using the asymmetric structure of claim 1, wherein the at least one processor is configured to:
increase a learning speed of an added adapter by adjusting an update of parameters learned prior to a reference point at a predetermined ratio using a Gradient Decoupling Layer (GDL).

### Claim 5 (depends on 1)

The continual learning apparatus using the asymmetric structure of claim 1, wherein the at least one processor is configured to:
store data samples from tasks prior to a reference point in the memory, and
perform replay based on the data samples.

### Claim 6 (depends on 1)

The continual learning apparatus using the asymmetric structure of claim 1, wherein the at least one processor is configured to:
determine a cross-entropy loss between the input text and a corresponding task identifier (ID), and
train the router to assign the input text to the adapter based on the cross-entropy loss.

### Claim 7

A method of a continual learning using an asymmetric structure, the method comprising:
storing a dataset, model parameters, a router, and adapters used for continual learning;
performing the continual learning using a neural network model comprising a shallow layer and a deep layer;
wherein the performing the continual learning comprises:
adding an adapter corresponding to new learning to the deep layer whenever new data is learned; and
distributing input text to at least one of the shallow layer, the deep layer, and the adapter via the router provided between the shallow layer and the deep layer,
performing regularization on the adapter to improve learning efficiency of the neural network model;
performing the regularization using an orthogonal loss function to maintain independence of information between different adapters;
performing the regularization using the orthogonal loss function by calculating orthogonality between output data of different adapters and projecting the output data onto different planes;
performing Sparse Low-rank Adaptation (SoRA) based on a gate vector provided between the adapters; and
determining a capacity of each of the adapters based on characteristics of the input text,
wherein the gate vector is composed of a vector in a rank dimension,
wherein the gate vector adjusts the rank by performing a Hadamard product with a feature vector that has passed through a predetermined layer,
wherein the orthogonal loss function is determined by Equation 1 as follows:
ℒ
=
∑
i
<
j
A
i
·
A
j
[
Equation
⁢
1
]
wherein L denotes a loss function, and Ai and Aj denote an i-th and a j-th A adapter, respectively,
wherein the Hadamard product is performed based on Equation 2 as follows:
h
=
A
·
(
g
⊙
(
B
·
x
)
)
[
Equation
⁢
2
]
wherein h represents a forward pass of each adapter, A and B represent parameters constituting each adapter, g represents the vector in the rank dimension, and x represents the feature vector that has passed through the predetermined layer.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
distributing the input text based on a feature corresponding to the deep layer via the router.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
increasing a learning speed of an added adapter by adjusting the update of parameters learned prior to a reference point at a predetermined ratio using a Gradient Decoupling Layer (GDL).

### Claim 10 (depends on 7)

The method of claim 7, further comprising:
determining a cross-entropy loss between the input text and a corresponding task identifier (ID), and
training the router to assign the input text to the adapter based on the cross-entropy loss.
