Method and apparatus for training natural language pre-trained model, device, and storage medium

An apparatus for training a natural language pre-trained model, comprising:

• a converting module configured to tokenize a text by using a dictionary of the natural language pre-trained model, and convert tokens in the text into one-hot codes;
• a mapping module configured to input the one-hot codes corresponding to the text into a token embedding layer, and perform mapping by using the token embedding layer to obtain a static token vector corresponding to each of the tokens;
• an input module configured to add the static token vector, a segment embedding vector, and a position embedding vector corresponding to each of the tokens to obtain a input vector of each of the tokens, and take the input vector as an input of the natural language pre-trained model to obtain a dynamic token vector corresponding to each of the tokens;
• a calculating module configured to calculate a similarity between the static token vector and the dynamic token vector corresponding to each of the tokens, and take a similarity calculation result as a constraint item;
• and
• an adjusting module configured to adjust an original loss function of the natural language pre-trained model by using the constraint item to obtain an adjusted original loss function, and train the natural language pre-trained model with the adjusted original loss function.