Method for automatically compressing multitask-oriented pre-trained language model and platform thereof

A method for method for automatic compressing multi-task oriented pre-trained language model, comprising the following three stages:

• a first stage of constructing a knowledge distillation coding vector based on Transformer layer sampling:
• layer-sampling all Transformer units of a BERT model by Bernoulli distribution to generate the knowledge distillation coding vector;
• a second stage of training a knowledge distillation network of meta-learning:
• defining a search space, inputting the knowledge distillation coding vector constructed in the first stage into the search space, and removing unqualified knowledge distillation coding vectors;
• defining a structure generator, which takes the filtered knowledge distillation coding vector as an input, outputs a weight matrix for constructing a distillation structure model, and generates the corresponding distillation structure model;
• training the generated distillation structure model to update the structure generator;
• a third stage of searching the distillation structure model based on an evolutionary algorithm:
• inputting a plurality of knowledge distillation coding vectors satisfying specific constraints into the updated structure generator in the second stage to generate the corresponding weight matrix, and obtaining a plurality of distillation structure models;
• evaluating the accuracy of each distillation structure model;
• using the evolutionary algorithm to search the distillation structure model with the highest accuracy that meets the specific constraints, and obtaining a common compression structure.