A method for compressing a pre-training language model based on knowledge distillation, wherein the method compresses a BERT model, which comprises a feature mapping knowledge distillation module, a self-attention cross knowledge distillation module and a linear learning module based on Bernoulli probability distribution;
- wherein, an original BERT model is a teacher model and a compressed BERT model is a student model;
- the feature mapping knowledge distillation module is based on a knowledge distillation strategy based on feature transfer;
- in a process of knowledge distillation from the teacher model to the student model, feature mapping of each layer of the student model is approaching feature mapping of the teacher model, and the student model focuses on intermediate layer features of the teacher model and uses the intermediate layer features to guide the student model;
- the self-attention cross knowledge distillation module realizes deep mutual learning between the teacher model and the student model by cross-connecting self-attention units of the teacher model and the student model by way of convex combination cross-connection on a network self-attention layer;
- the linear learning module based on Bernoulli probability distribution gradually completes feature mapping from the teacher model to the student model and knowledge transfer from self-attentive distribution;
- the self-attention cross knowledge distillation module comprises the following three stages:
- a first stage:
- the self-attention unit of a teacher network is input into a student network, and a transfer objective function is minimized;
- specifically, the self-attention unit of the teacher network is regarded as a basic truth value, the student network is input at the self-attention unit of the network, and the student network receives a correct supervision signal to train a subsequent layer, avoiding a phenomenon of an excessive estimation error and propagation thereof;
- a second stage:
- the self-attention unit of the student network is input into the teacher network and the transfer objective function is minimized;
- because the estimation error propagates layer by layer on the student network, there is a difference between an input of the student network and an input of the teacher network on a same layer;
- the self-attention unit of the student network is input to the teacher network, which makes the student network imitate an output behavior of the teacher network on the premise of a same input;
- a third stage:
- migration objective functions of the first stage and the second stage on the self-attention unit of the network are convexly combined to realize the distillation strategy of cross migration;
- the linear learning module based on Bernoulli probability distribution is configured to set different linear migration probabilities for driving the feature mapping knowledge distillation module and the self-attention cross knowledge distillation module, and comprises the following two steps:
- step 1, the feature mapping knowledge distillation module and the self-attention cross knowledge distillation module both adopt the migration probability of Bernoulli probability distribution, that is, assuming that an ith module is currently migrated, a random variable X is sampled through a Bernoulli distribution, and X is 0 or 1;
- the random variable being 1 represents that transfer learning is performed for a current module, and the random variable not being 1 represents that transfer learning is not performed;
- step 2:
- although setting a constant migration probability p in step 1 can meet the needs of a compression model, a linear learning-driven migration probability is helpful to gradually migrate an encoder module in the model;
- this step designs a linear learning-driven migration probability plinear to dynamically adjust the migration probability p in step 1, namely
- plinear=min(1, k*i+b)
- where, plinear represents the migration probability of the current migration module, migration of the ith module corresponds to an ith step of a current training, and b represents an initial migration probability without training;
- k is a dynamic value greater than 0, and when the training is increased to 1000 steps, 5000 steps, 10000 steps and 30000 steps, plinear is gradually increased to 0.25, 0.5, 0.75 and 1.00 in turn.