Method and apparatus for training text prediction model, and text prediction method and apparatus

A method for training a text prediction model, comprising:

• obtaining a training data set, the training data set comprising input text samples and sentiment category labels corresponding to the input text samples;
• and
• training the text prediction model by using a text sequence comprising an input text sample and a sentiment category label corresponding to the input text sample as an input of the text prediction model, wherein the text prediction model is a large language model, and the large language model comprises a plurality of layers of concatenated Transformer networks;
• and
• in the training, each Transformer network is used as a current Transformer network layer to execute:
• determining, in a current round of iteration, a first key matrix, a first value matrix, and a first query matrix by using a feature representation outputted by a previous network layer;
• splicing the first key matrix and a second key matrix that is obtained by the current Transformer network layer in a previous round of iteration, to obtain a third key matrix, and splicing the first value matrix and a second value matrix that is obtained by the current Transformer network layer in the previous round of iteration, to obtain a third value matrix;
• performing processing of a self-attention mechanism by using the third key matrix, the third value matrix, and the first query matrix, to obtain a feature representation outputted by the current Transformer network layer;
• updating the second key matrix by using the first key matrix, and determining an updated second key matrix as a second key matrix that is obtained by the current Transformer network layer in the current round of iteration;
• and updating the second value matrix by using the first value matrix, and determining an updated second value matrix as a second value matrix that is obtained by the current Transformer network layer in the current round of iteration.

An electronic device, comprising:

• at least one processor;
• and
• a memory associated with the at least one processor, wherein the memory is configured to store program instructions, and the program instructions, when read and executed by the at least one processor, performs the following steps:
• obtaining a training data set, the training data set comprising input text samples and output labels corresponding to the input text samples;
• training the text prediction model by using a text sequence comprising an input text sample and an output label corresponding to the input text sample as an input of the text prediction model, wherein the text prediction model is a large language model, and the large language model comprises a plurality of layers of concatenated Transformer networks;
• and
• in the training, each Transformer network is used as a current Transformer network layer to execute:
• determining, in a current round of iteration, a first key matrix, a first value matrix, and a first query matrix by using a feature representation outputted by a previous network layer;
• splicing the first key matrix and a second key matrix that is obtained by the current Transformer network layer in a previous round of iteration, to obtain a third key matrix, and splicing the first value matrix and a second value matrix that is obtained by the current Transformer network layer in the previous round of iteration, to obtain a third value matrix;
• performing processing of a self-attention mechanism by using the third key matrix, the third value matrix, and the first query matrix, to obtain a feature representation outputted by the current Transformer network layer;
• updating the second key matrix by using the first key matrix, and determining an updated second key matrix as a second key matrix that is obtained by the current Transformer network layer in the current round of iteration;
• and updating the second value matrix by using the first value matrix, and determining an updated second value matrix as a second value matrix that is obtained by the current Transformer network layer in the current round of iteration.