# US 2026/0087254 A1 · Method for merging language models and apparatus for implementing the same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087254 A1
- Document ID: US20260087254A1
- Status: Published application
- Kind: A1
- Application number: 19/264,073
- Filed: 2025-07-09
- Published: 2026-03-26
- Earliest claimed: 2024-09-23
- Assignees: Samsung Sds Co., Ltd.
- Applicants: Samsung Sds Co., Ltd.
- Inventors: Ho Young KANG; Bong Kyu HWANG; Soo Ah CHO; Jun Hwa CHOI; Seong Ho JOE
- Web page: https://patentgrep.com/patent/US20260087254A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure according to at least one embodiment provides a method for merging language models, performed by a computing system. The method comprises in response to receipt of a request to merge a first language model and a second language model, converting first embedding vectors corresponding to a first tokenizer used in the first language model and second embedding vectors corresponding to a second tokenizer used in the second language model by merging the first tokenizer and the second tokenizer, wherein the second tokenizer is different from the first tokenizer, repeatedly reducing components of each of the converted first embedding vectors and converted second embedding vectors through Singular Value Decomposition (SVD) until a preset performance threshold is reached, and merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

## Claims

### Claim 1

A method for merging language models, performed by a computing system, the method comprising:
in response to receipt of a request to merge a first language model and a second language model, converting first embedding vectors corresponding to a first tokenizer used in the first language model and second embedding vectors corresponding to a second tokenizer used in the second language model by merging the first tokenizer and the second tokenizer, wherein the second tokenizer is different from the first tokenizer;
repeatedly reducing components of each of the converted first embedding vectors and converted second embedding vectors through Singular Value Decomposition (SVD) until a preset performance threshold is reached; and
merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

### Claim 2 (depends on 1)

The method of claim 1, wherein the converting of the first embedding vectors corresponding to the first tokenizer and the second embedding vectors corresponding to the second tokenizer comprises: merging a first vocabulary list of the first tokenizer and a second vocabulary list of the second tokenizer; expanding a dimension of the first embedding vectors and a dimension of the second embedding vectors to match whichever of the two dimensions is larger; tokenizing, via the first tokenizer, first added vocabulary items added to the first vocabulary list through the merging of the first and second vocabulary lists, and initializing the tokenized first added vocabulary items with an average value of corresponding embeddings; and tokenizing, via the second tokenizer, second added vocabulary items added to the second vocabulary list through the merging of the first and second vocabulary lists, and initializing the tokenized second added vocabulary items with an average value of corresponding embeddings.

### Claim 3 (depends on 1)

The method of claim 1, wherein
the first embedding vectors include a first input embedding vector that delivers an input value to a first layer among a plurality of layers of the first language model and a first output embedding vector that receives an output value from a last layer of the first language model, and
the second embedding vectors include a second input embedding vector that delivers an input value to a first layer among a plurality of layers of the second language model and a second output embedding vector that receives an output value from a last layer of the second language model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the repeatedly reducing of the components of each of the converted first embedding vectors and converted second embedding vectors comprises: obtaining bases corresponding to each of the first and second input embedding vectors by performing SVD on matrices of the first layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second input embedding vectors; terminating the reducing of the components of each embedding vector when performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first input embedding vector and the reduced second input embedding vector.

### Claim 5 (depends on 4)

The method of claim 4, wherein the repeatedly reducing of the components of each of the converted first embedding vectors and converted second embedding vectors comprises: obtaining bases corresponding to each of the first and second output embedding vectors by performing SVD on matrices of the last layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second output embedding vectors; terminating the reducing of the components of each embedding vector when the performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first output embedding vector and the reduced second output embedding vector.

### Claim 6 (depends on 5)

The method of claim 5, wherein the merging of the first and second language models comprises: updating the first language model using the reduced first input and output embedding vectors; updating the second language model using the reduced second input and output embedding vectors; and merging the updated first and second language models.

### Claim 7

A method for merging language models, performed by a computing system, the method comprising:
merging a first vocabulary list of a first tokenizer used in a first language model and a second vocabulary list of a second tokenizer used in a second language model, wherein the second tokenizer is different from the first tokenizer;
obtaining first embedding vectors corresponding to the first tokenizer and second embedding vectors corresponding to the second tokenizer using the merged first and second vocabulary lists;
reducing components of each of the first embedding vectors and second embedding vectors until a preset performance threshold is reached; and
merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

### Claim 8 (depends on 7)

The method of claim 7, wherein the obtaining of the first embedding vectors corresponding to the first tokenizer and the second embedding vectors corresponding to the second tokenizer comprises: tokenizing, via the first tokenizer, first added vocabulary items added to the first vocabulary list of the first tokenizer through the merging of the first and second vocabulary lists, and initializing the tokenized first added vocabulary items with an average value of corresponding embeddings; and tokenizing, via the second tokenizer, second added vocabulary items added to the second vocabulary list of the second tokenizer through the merging of the first and second vocabulary lists, and initializing the tokenized second added vocabulary items with an average value of corresponding embeddings.

### Claim 9 (depends on 7)

The method of claim 7, wherein
the first embedding vectors include a first input embedding vector that delivers an input value to a first layer among a plurality of layers of the first language model and a first output embedding vector that receives an output value from a last layer of the first language model, and
the second embedding vectors include a second input embedding vector that delivers an input value to a first layer among a plurality of layers of the second language model and a second output embedding vector that receives an output value from a last layer of the second language model.

### Claim 10 (depends on 9)

The method of claim 9, wherein the reducing of the components of each of the first and second embedding vectors comprises: obtaining bases corresponding to each of the first and second input embedding vectors by performing SVD on matrices of the first layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second input embedding vectors; terminating the reducing of the components of each embedding vector when a performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first input embedding vector and the reduced second input embedding vector.

### Claim 11 (depends on 10)

The method of claim 10, wherein the merging of the first and second language models comprises: updating the first language model using the reduced first input and output embedding vectors; updating the second language model using the reduced second input and output embedding vectors; and merging the updated first and second language models.

### Claim 12

A system for merging language models, comprising:
at least one processor;
a memory configured to load a computer program executed by the at least one processor; and
a storage configured to store the computer program,
wherein the computer program includes instructions for performing operations of: in response to receipt of a request to merge a first language model and a second language model, converting first embedding vectors corresponding to a first tokenizer used in the first language model and second embedding vectors corresponding to a second tokenizer used in the second language model by merging the first tokenizer and the second tokenizer, wherein the second tokenizer is different from the first tokenizer; repeatedly reducing components of each of the converted first embedding vectors and converted second embedding vectors through Singular Value Decomposition (SVD) until a preset performance threshold is reached; and merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operation of converting the first embedding vectors corresponding to the first tokenizer and the second embedding vectors corresponding to the second tokenizer comprises: merging a first vocabulary list of the first tokenizer and a second vocabulary list of the second tokenizer; expanding a dimension of the first embedding vectors and a dimension of the second embedding vectors to match whichever of the two dimensions is larger; tokenizing, via the first tokenizer, first added vocabulary items added to the first vocabulary list through the merging of the first and second vocabulary lists, and initializing the tokenized first added vocabulary items with an average value of corresponding embeddings; and tokenizing, via the second tokenizer, second added vocabulary items added to the second vocabulary list through the merging of the first and second vocabulary lists, and initializing the tokenized second added vocabulary items with an average value of corresponding embeddings.

### Claim 14 (depends on 12)

The system of claim 12, wherein
the first embedding vectors include a first input embedding vector that delivers an input value to a first layer among a plurality of layers of the first language model and a first output embedding vector that receives an output value from a last layer of the first language model, and
the second embedding vectors include a second input embedding vector that delivers an input value to a first layer among a plurality of layers of the second language model and a second output embedding vector that receives an output value from a last layer of the second language model.

### Claim 15 (depends on 14)

The system of claim 14, wherein the operation of repeatedly reducing the components of each of the converted first embedding vectors and converted second embedding vectors comprises: obtaining bases corresponding to each of the first and second input embedding vectors by performing SVD on matrices of the first layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second input embedding vectors; terminating the reducing of the components of each embedding vector when a performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first input embedding vector and the reduced second input embedding vector.

### Claim 16 (depends on 15)

The system of claim 15, wherein the operation of repeatedly reducing the components of each of the converted first embedding vectors and converted second embedding vectors comprises: obtaining bases corresponding to each of the first and second output embedding vectors by performing SVD on matrices of the last layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second output embedding vectors; terminating the reducing of the components of each embedding vector when the performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first output embedding vector and the reduced second output embedding vector.

### Claim 17 (depends on 16)

The system of claim 16, wherein the operation of merging the first and second language models comprises: updating the first language model using the reduced first input and output embedding vectors; updating the second language model using the reduced second input and output embedding vectors; and merging the updated first and second language models.

### Claim 18

A system for merging language models, comprising:
at least one processor;
a memory configured to load a computer program executed by the at least one processor; and
a storage configured to store the computer program,
wherein the computer program includes instructions for performing operations of: merging a first vocabulary list of a first tokenizer used in a first language model and a second vocabulary list of a second tokenizer used in a second language model, wherein the second tokenizer is different from the first tokenizer; obtaining first embedding vectors corresponding to the first tokenizer and second embedding vectors corresponding to the second tokenizer using the merged first and second vocabulary lists; reducing components of each of the first embedding vectors and second embedding vectors until a preset performance threshold is reached; and merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

### Claim 19 (depends on 18)

The system of claim 18, wherein
the first embedding vectors include a first input embedding vector that delivers an input value to a first layer among a plurality of layers of the first language model and a first output embedding vector that receives an output value from a last layer of the first language model, and
the second embedding vectors include a second input embedding vector that delivers an input value to a first layer among a plurality of layers of the second language model and a second output embedding vector that receives an output value from a last layer of the second language model.

### Claim 20 (depends on 19)

The system of claim 19, wherein the reducing of the components of each of the first and second embedding vectors comprises: obtaining bases corresponding to each of the first and second input embedding vectors by performing SVD on matrices of the first layers of the first and second language models; reducing components of each embedding vector by reducing a number of bases corresponding to each of the first and second input embedding vectors; terminating the reducing of the components of each embedding vector when a performance of each of the first and second language models reaches the preset performance threshold; and obtaining the reduced first input embedding vector and the reduced second input embedding vector.
