# US 2025/0061305 A1 · Training method for a deep learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061305 A1
- Document ID: US20250061305A1
- Status: Published application
- Kind: A1
- Application number: 18/936,686
- Filed: 2024-11-04
- Published: 2025-02-20
- Earliest claimed: 2023-12-20
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science Technology Co., Ltd.
- Inventors: Shuohuan WANG; Junyuan SHANG; Yinqi YANG; Guoxia WANG; Linhao ZHANG; Yu SUN; Hua WU; Haifeng WANG
- Web page: https://patentgrep.com/patent/US20250061305A1

## CPC

- G06N 3/043: based on fuzzy logic, fuzzy membership or fuzzy inference, e.g. adaptive neuro-fuzzy inference systems [ANFIS]
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A training method, an inference method, a device, an apparatus, and a medium for a deep learning model are provided. A first model includes a plurality of first parameters, a second model comprises a plurality of second parameters, which is initialized to parameter values of a plurality of target parameters selected from the plurality of first parameters. The training method includes: determining a target loss for both the first model and the second model; adjusting parameter values, including: in response to determining that the target loss indicates that the parameter values of at least part of the target parameters need to be adjusted, synchronously adjusting the parameter values of the corresponding second parameters; and in response to determining that the target loss indicates that the parameter values of at least part of the second parameters need to be adjusted, synchronously adjusting the parameter values of the corresponding target parameters.

## Claims

### Claim 1

A training method for a deep learning model, wherein a first deep learning model comprises a plurality of first parameters, a second deep learning model comprises a plurality of second parameters, a number of the plurality of second parameters is less than a number of the plurality of first parameters, the plurality of first parameters includes a plurality of target parameters corresponding to the plurality of the second parameters, wherein the plurality of the second parameters are initialized to parameter values of the plurality of target parameters, the method comprises:
determining a target loss for both the first deep learning model and the second deep learning model;
adjusting parameter values of the plurality of first parameters and parameter values of the plurality of second parameters based on the target loss to obtain a trained first deep learning model and a trained second deep learning model, comprising:
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of target parameters included in the first deep learning model need to be adjusted, synchronously adjusting parameter values of second parameters included in the second deep learning model corresponds to the at least part of the target parameters; and
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of second parameters included in the second deep learning model need to be adjusted, synchronously adjusting parameter values of target parameters included in the first deep learning model corresponds to the at least part of the second parameters.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the target loss for both the first deep learning model and the second deep learning model comprises:
determining a first loss of the first deep learning model for a first sample;
determining a second loss of the second deep learning model for a second sample; and
determining the target loss based on the first loss and the second loss.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first loss is the same as the second loss.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first deep learning model includes a first layer quantity of first layers, each first layer of the first layers includes a first intermediate dimension quantity of first intermediate dimensions,
the second deep learning model includes a second layer quantity of second layers, each second layer of the second layers includes a second intermediate dimension quantity of second intermediate dimensions, wherein the second layer quantity is less than the first layer quantity, and the second intermediate dimension quantity is less than the first intermediate dimension quantity,
the plurality of target parameters include first parameters in at least one target intermediate dimension of each target layer of at least one target layer, wherein the at least one target layer is obtained by selecting in the first layer quantity of first layers based on the second layer quantity, and the at least one target intermediate dimension is obtained by selecting in the first intermediate dimension quantity of first intermediate dimensions based on the second intermediate dimension quantity.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first deep learning model and the second deep learning model are based on a Transformer structure or a Transformer-like structure,
the first deep learning model includes a first hidden dimension quantity of first hidden dimensions, and each first layer of the first layers includes a first attention head quantity of first attention heads,
the second deep learning model includes a second hidden dimension quantity of second hidden dimensions, and each second layer of the second layers includes a second attention head quantity of second attention heads,
the plurality of target parameters include first parameters corresponding to a plurality of target hidden dimensions, and the plurality of target parameters include first parameters in at least one target attention head, wherein the plurality of target hidden dimensions is obtained by selecting in the first hidden dimension quantity of first hidden dimensions based on the second hidden dimension quantity, and the at least one target attention head is obtained by selecting in the first attention head quantity of first attention heads based on the second attention head quantity.

### Claim 6 (depends on 5)

The method of claim 5, wherein the at least one target layer, the at least one target intermediate dimension, the plurality of target hidden dimensions, and/or the at least one target attention head are obtained by random selection.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first deep learning model is a pre-trained large model.

### Claim 8 (depends on 1)

The method of claim 1, wherein both the first deep learning model and the second deep learning model are configured to perform at least one of the following tasks: a text processing task, an image processing task, and a speech processing task.

### Claim 9

An inference method for a deep learning model, wherein a first deep learning model comprises a plurality of first parameters, a second deep learning model comprises a plurality of second parameters, a number of the plurality of second parameters is less than a number of the plurality of first parameters, the plurality of first parameters includes a plurality of target parameters corresponding to the plurality of the second parameters, wherein the plurality of the second parameters are initialized to parameter values of the plurality of target parameters, the first deep learning model and the second learning model are trained by performing training operations, the training operations comprises:
determining a target loss for both the first deep learning model and the second deep learning model;
adjusting parameter values of the plurality of first parameters and parameter values of the plurality of second parameters based on the target loss to obtain a trained first deep learning model and a trained second deep learning model, comprising:
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of target parameters included in the first deep learning model need to be adjusted, synchronously adjusting parameter values of second parameters included in the second deep learning model corresponds to the at least part of the target parameters; and
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of second parameters included in the second deep learning model need to be adjusted, synchronously adjusting parameter values of target parameters included in the first deep learning model corresponds to the at least part of the second parameters,
the inference method comprises:
generating a plurality of predicted tokens sequentially using the trained second deep learning model;
generating confidence level for each predicted token of the plurality of predicted tokens based on the plurality of predicted tokens using the trained first deep learning model; and
verifying the plurality of predicted tokens based on a generation sequence of the plurality of predicted tokens to obtain an inference result, comprising:
in response to determining that the confidence level of a currently verified predicted token is lower than a preset threshold, generating a correction token at a position of the currently verified predicted token using the trained first deep learning model to replace the currently verified predicted token as a verified predicted token;
in response to determining that a preset token generation condition is satisfied, regenerating predicted tokens sequentially from a next position of the correction token based on the correction token using the trained second deep learning model and generating a confidence level for each regenerated predicted token of the regenerated predicted tokens using the trained first deep learning model for verification; and
obtaining the inference result based on the verified predicted tokens.

### Claim 10 (depends on 9)

The method of claim 9, wherein the verifying the plurality of predicted tokens comprises:
in response to determining that the confidence level of the currently verified predicted token is not less than the preset threshold, retaining the predicted token as a verified predicted token.

### Claim 11 (depends on 9)

The method of claim 9, wherein the preset token generation condition includes that the number of generated tokens is less than a preset number.

### Claim 12 (depends on 9)

The method of claim 9, wherein the preset token generation condition includes the correction token indicating that token generation has not ended.

### Claim 13

A electronic device, comprising:
one or more processors;
a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
a deep learning model, wherein a first deep learning model comprises a plurality of first parameters, a second deep learning model comprises a plurality of second parameters, a number of the plurality of second parameters is less than a number of the plurality of first parameters, the plurality of first parameters includes a plurality of target parameters corresponding to the plurality of the second parameters, wherein the plurality of the second parameters are initialized to parameter values of the plurality of target parameters;
determining a target loss for both the first deep learning model and the second deep learning model;
adjusting parameter values of the plurality of first parameters and parameters values of the plurality of second parameters based on the target loss to obtain a trained first deep learning model and a trained second deep learning model, comprising:
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of target parameters included in the first deep learning model need to be adjusted, synchronously adjusting parameter values of second parameters included in the second deep learning model corresponds to the at least part of the target parameters; and
in response to determining that the target loss indicates that the parameter values of at least part of the plurality of second parameters included in the second deep learning model need to be adjusted, synchronously adjust parameter values of target parameters included in the first deep learning model corresponds to the at least part of the second parameters.

### Claim 14 (depends on 13)

The electronic device of claim 13, wherein determining the target loss for both the first deep learning model and the second deep learning model comprises:
determining a first loss of the first deep learning model for a first sample;
determining a second loss of the second deep learning model for a second sample; and
determining the target loss based on the first loss and the second loss.

### Claim 15 (depends on 14)

The electronic device of claim 14, wherein the first sample is the same as the second sample.

### Claim 16 (depends on 13)

The electronic device of claim 13, wherein the first deep learning model includes a first layer quantity of first layers, each first layer of the first layers includes a first intermediate dimension quantity of first intermediate dimensions,
the second deep learning model includes a second layer quantity of second layers, each second layer of the second layers includes a second intermediate dimension quantity of second intermediate dimensions, wherein the second layer quantity is less than the first layer quantity, and the second intermediate dimension quantity is less than the first intermediate dimension quantity,
the plurality of target parameters include first parameters in at least one target intermediate dimension of each target layer of at least one target layer, wherein the at least one target layer is obtained by selecting in the first layer quantity of first layers based on the second layer quantity, and the at least one target intermediate dimension is obtained by selecting in the first intermediate dimension quantity of first intermediate dimensions based on the second intermediate dimension quantity.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein the first deep learning model and the second deep learning model are based on a Transformer structure or a Transformer-like structure,
the first deep learning model includes a first hidden dimension quantity of first hidden dimensions, and each first layer of the first layers includes a first attention head quantity of first attention heads,
the second deep learning model includes a second hidden dimension quantity of second hidden dimensions, and each second layer of the second layers includes a second attention head quantity of second attention heads,
the plurality of target parameters include first parameters corresponding to a plurality of target hidden dimensions, and the plurality of target parameters include first parameters in at least one target attention head, wherein the plurality of target hidden dimensions are obtained by selecting in the first hidden dimension quantity of first hidden dimensions based on the second hidden dimension quantity, and the at least one target attention head is obtained by selecting in the first attention head quantity of first attention heads based on the second attention head quantity.

### Claim 18 (depends on 17)

The electronic device of claim 17, wherein the at least one target layer, the at least one target intermediate dimension, the plurality of target hidden dimensions, and/or the at least one target attention head are obtained by random selection.

### Claim 19 (depends on 13)

The electronic device of claim 13, wherein the first deep learning model is a pre-trained large model.

### Claim 20 (depends on 13)

The electronic device of claim 13, wherein both the first deep learning model and the second deep learning model are configured to perform at least one of the following tasks: a text processing task, an image processing task, and a speech processing task.
