# US 2025/0094806 A1 · Large language model training method, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094806 A1
- Document ID: US20250094806A1
- Status: Published application
- Kind: A1
- Application number: 18/967,167
- Filed: 2024-12-03
- Published: 2025-03-20
- Earliest claimed: 2024-05-14
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Junyuan Shang; Yilong Chen; Zhenyu Zhang; Shuohuan Wang; Yu Sun; Hua Wu
- Web page: https://patentgrep.com/patent/US20250094806A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Provided is a large language model training method, an electronic device and a storage medium, relating to the field of artificial intelligence technologies, and in particular, to the fields of deep learning, natural language processing and large model. The method includes: performing dimension reduction parameter fusion on a two-dimensional parameter matrix on each channel in each network layer in a first large language model, respectively, to obtain a second large language model; performing layer reduction parameter fusion on network layers in the second large language model based on a three-dimensional parameter matrix of each network layer in the second large language model to obtain a third large language model; and training the third large language model to obtain a target large language model under the condition that the target loss function determined based on the first and third large language models meets a preset first function condition.

## Claims

### Claim 1

A large language model training method, comprising:
performing dimension reduction parameter fusion on a two-dimensional parameter matrix on each channel in each network layer in a first large language model, respectively, to obtain a second large language model;
performing layer reduction parameter fusion on network layers in the second large language model based on a three-dimensional parameter matrix of each network layer in the second large language model to obtain a third large language model;
determining a target loss function based on the first large language model and the third large language model; and
training the third large language model to obtain a target large language model after determining that the target loss function meets a preset first function condition.

### Claim 2 (depends on 1)

The method of claim 1, wherein the performing the dimension reduction parameter fusion comprises:
decomposing a dimension fusion operator to obtain a column conversion matrix and a row conversion matrix corresponding to each channel in each network layer in the first large language model;
performing column-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the first large language model, respectively, based on the column conversion matrix corresponding to each channel in each network layer in the first large language model to obtain a fourth large language model; and
performing row-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the fourth large language model, respectively, based on the row conversion matrix corresponding to each channel in each network layer in the first large language model to obtain the second large language model.

### Claim 3 (depends on 2)

The method of claim 2, wherein the performing the column-wise dimension reduction parameter fusion comprises:
for a first column conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a first difference value between a row number of the first column conversion matrix and a column number of a corresponding two-dimensional parameter matrix in the first large language model meets a first difference value condition, updating the first column conversion matrix based on a ratio of the row number of the first column conversion matrix to a preset column receptive field value; and
multiplying the updated first column conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the column-wise dimension reduction parameter fusion,
wherein the preset column receptive field value is more than 1.

### Claim 4 (depends on 2)

The method of claim 2, wherein the performing the row-wise dimension reduction parameter fusion comprises:
for a first row conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a second difference value between a row number of the first row conversion matrix and a row number of a corresponding two-dimensional parameter matrix in the first large language model meets a second difference value condition, updating the first row conversion matrix based on a ratio of the row number of the first row conversion matrix to a preset row receptive field value; and
multiplying the updated first row conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the row-wise dimension reduction parameter fusion,
wherein the preset row receptive field value is more than 1.

### Claim 5 (depends on 2)

The method of claim 2, further comprising:
adjusting the dimension fusion operator based on the target loss function after determining that the target loss function does not meet the preset first function condition.

### Claim 6 (depends on 1)

The method of claim 1, wherein the performing the layer reduction parameter fusion on the network layers in the second large language model comprises:
determining a mapping coefficient from each network layer in a first model to each network layer in a second model based on a layer fusion operator, wherein a number of network layers in the first model is equal to a number of network layers in the second large language model;
for each second network layer in the second model, obtaining a three-dimensional parameter matrix of the second network layer based on the three-dimensional parameter matrix of each network layer in the second large language model and the mapping coefficient from each network layer in the first model to the second network layer, respectively; and
determining the third large language model based on the three-dimensional parameter matrix of each second network layer.

### Claim 7 (depends on 6)

The method of claim 6, wherein the obtaining the three-dimensional parameter matrix of the second network layer comprises:
multiplying the three-dimensional parameter matrix of each network layer in the second large language model by the mapping coefficient from a corresponding network layer in the first model to the second network layer, respectively, to obtain a plurality of three-dimensional parameter matrices; and
summing the plurality of three-dimensional parameter matrices to obtain the three-dimensional parameter matrix of the second network layer.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
adjusting the layer fusion operator based on the target loss function after determining that the target loss function does not meet the preset first function condition.

### Claim 9 (depends on 1)

The method of claim 1, wherein the determining the target loss function based on the first large language model and the third large language model comprises:
determining a first loss function and a first output distribution of the first large language model based on a first training sample set;
determining a second loss function and a second output distribution of the third large language model based on a second training sample set;
determining a third loss function based on the first output distribution and the second output distribution; and
performing weighted summation on the first loss function, the second loss function and the third loss function to obtain the target loss function.

### Claim 10

An electronic device, comprising:
at least one processor; and
a memory connected in communication with the at least one processor,
wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, enables the at least one processor to execute:
performing dimension reduction parameter fusion on a two-dimensional parameter matrix on each channel in each network layer in a first large language model, respectively, to obtain a second large language model;
performing layer reduction parameter fusion on network layers in the second large language model based on a three-dimensional parameter matrix of each network layer in the second large language model to obtain a third large language model;
determining a target loss function based on the first large language model and the third large language model; and
training the third large language model to obtain a target large language model after determining that the target loss function meets a preset first function condition.

### Claim 11 (depends on 10)

The electronic device of claim 10, wherein the instruction, when executed by the at least one processor, enables the at least one processor to execute: performing the dimension reduction parameter fusion by:
decomposing a dimension fusion operator to obtain a column conversion matrix and a row conversion matrix corresponding to each channel in each network layer in the first large language model;
performing column-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the first large language model, respectively, based on the column conversion matrix corresponding to each channel in each network layer in the first large language model to obtain a fourth large language model; and
performing row-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the fourth large language model, respectively, based on the row conversion matrix corresponding to each channel in each network layer in the first large language model to obtain the second large language model.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the instruction, when executed by the at least one processor, enables the at least one processor to execute: performing the column-wise dimension reduction parameter fusion by:
for a first column conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a first difference value between a row number of the first column conversion matrix and a column number of a corresponding two-dimensional parameter matrix in the first large language model meets a first difference value condition, updating the first column conversion matrix based on a ratio of the row number of the first column conversion matrix to a preset column receptive field value; and
multiplying the updated first column conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the column-wise dimension reduction parameter fusion,
wherein the preset column receptive field value is more than 1.

### Claim 13 (depends on 11)

The electronic device of claim 11, wherein the instruction, when executed by the at least one processor, enables the at least one processor to execute: performing the row-wise dimension reduction parameter fusion by:
for a first row conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a second difference value between a row number of the first row conversion matrix and a row number of a corresponding two-dimensional parameter matrix in the first large language model meets a second difference value condition, updating the first row conversion matrix based on a ratio of the row number of the first row conversion matrix to a preset row receptive field value; and
multiplying the updated first row conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the row-wise dimension reduction parameter fusion,
wherein the preset row receptive field value is more than 1.

### Claim 14 (depends on 11)

The electronic device of claim 11, wherein the instruction, when executed by the at least one processor, enables the at least one processor to further execute:
adjusting the dimension fusion operator based on the target loss function after determining that the target loss function does not meet the preset first function condition.

### Claim 15 (depends on 10)

The electronic device of claim 10, wherein the instruction, when executed by the at least one processor, enables the at least one processor to execute: performing the layer reduction parameter fusion by:
determining a mapping coefficient from each network layer in a first model to each network layer in a second model based on a layer fusion operator, wherein a number of network layers in the first model is equal to a number of network layers in the second large language model;
for each second network layer in the second model, obtaining a three-dimensional parameter matrix of the second network layer based on the three-dimensional parameter matrix of each network layer in the second large language model and the mapping coefficient from each network layer in the first model to the second network layer, respectively; and
determining the third large language model based on the three-dimensional parameter matrix of each second network layer.

### Claim 16

A non-transitory computer readable storage medium storing a computer instruction wherein the computer instruction causes a computer to perform:
performing dimension reduction parameter fusion on a two-dimensional parameter matrix on each channel in each network layer in a first large language model, respectively, to obtain a second large language model;
performing layer reduction parameter fusion on network layers in the second large language model based on a three-dimensional parameter matrix of each network layer in the second large language model to obtain a third large language model;
determining a target loss function based on the first large language model and the third large language model; and
training the third large language model to obtain a target large language model after determining that the target loss function meets a preset first function condition.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein the computer instruction causes the computer to perform the dimension reduction parameter fusion by:
decomposing a dimension fusion operator to obtain a column conversion matrix and a row conversion matrix corresponding to each channel in each network layer in the first large language model;
performing column-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the first large language model, respectively, based on the column conversion matrix corresponding to each channel in each network layer in the first large language model to obtain a fourth large language model; and
performing row-wise dimension reduction parameter fusion on the two-dimensional parameter matrix on each channel in each network layer in the fourth large language model, respectively, based on the row conversion matrix corresponding to each channel in each network layer in the first large language model to obtain the second large language model.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the computer instruction causes the computer to perform the column-wise dimension reduction parameter fusion by:
for a first column conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a first difference value between a row number of the first column conversion matrix and a column number of a corresponding two-dimensional parameter matrix in the first large language model meets a first difference value condition, updating the first column conversion matrix based on a ratio of the row number of the first column conversion matrix to a preset column receptive field value; and
multiplying the updated first column conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the column-wise dimension reduction parameter fusion,
wherein the preset column receptive field value is more than 1.

### Claim 19 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the computer instruction causes the computer to perform the row-wise dimension reduction parameter fusion by:
for a first row conversion matrix corresponding to a first channel in a first network layer in the first large language model, after determining that a second difference value between a row number of the first row conversion matrix and a row number of a corresponding two-dimensional parameter matrix in the first large language model meets a second difference value condition, updating the first row conversion matrix based on a ratio of the row number of the first row conversion matrix to a preset row receptive field value; and
multiplying the updated first row conversion matrix by the corresponding two-dimensional parameter matrix in the first large language model to perform the row-wise dimension reduction parameter fusion,
wherein the preset row receptive field value is more than 1.

### Claim 20 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein the computer instruction causes the computer to further perform:
adjusting the dimension fusion operator based on the target loss function after determining that the target loss function does not meet the preset first function condition.
