Method for merging language models and apparatus for implementing the same

A method for merging language models, performed by a computing system, the method comprising:

• merging a first vocabulary list of a first tokenizer used in a first language model and a second vocabulary list of a second tokenizer used in a second language model, wherein the second tokenizer is different from the first tokenizer;
• obtaining first embedding vectors corresponding to the first tokenizer and second embedding vectors corresponding to the second tokenizer using the merged first and second vocabulary lists;
• reducing components of each of the first embedding vectors and second embedding vectors until a preset performance threshold is reached;
• and
• merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

A system for merging language models, comprising:

• at least one processor;
• a memory configured to load a computer program executed by the at least one processor;
• and
• a storage configured to store the computer program,
• wherein the computer program includes instructions for performing operations of:
• in response to receipt of a request to merge a first language model and a second language model, converting first embedding vectors corresponding to a first tokenizer used in the first language model and second embedding vectors corresponding to a second tokenizer used in the second language model by merging the first tokenizer and the second tokenizer, wherein the second tokenizer is different from the first tokenizer;
• repeatedly reducing components of each of the converted first embedding vectors and converted second embedding vectors through Singular Value Decomposition (SVD) until a preset performance threshold is reached;
• and merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.

A system for merging language models, comprising:

• at least one processor;
• a memory configured to load a computer program executed by the at least one processor;
• and
• a storage configured to store the computer program,
• wherein the computer program includes instructions for performing operations of:
• merging a first vocabulary list of a first tokenizer used in a first language model and a second vocabulary list of a second tokenizer used in a second language model, wherein the second tokenizer is different from the first tokenizer;
• obtaining first embedding vectors corresponding to the first tokenizer and second embedding vectors corresponding to the second tokenizer using the merged first and second vocabulary lists;
• reducing components of each of the first embedding vectors and second embedding vectors until a preset performance threshold is reached;
• and merging the first and second language models using the reduced first embedding vectors and the reduced second embedding vectors.