Method and apparatus for selecting sample corpus used to optimize translation model

An apparatus for selecting a sample corpus used to optimize a translation model, comprising:

• at least one processor;
• and
• a memory storing instructions, wherein the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
• step i):
• acquiring a first corpus, and translating the first corpus by using a to-be-optimized translation model to generate a second corpus corresponding to the first corpus, wherein a language type of the first corpus and a language type of the second corpus are different, and the to-be-optimized translation model is configured to perform bidirectional translation between the language type of the first corpus and the language type of the second corpus;
• step ii):
• translating the second corpus by using the to-be-optimized translation model to generate a third corpus corresponding to the second corpus, wherein a language type of the third corpus and the language type of the first corpus are the same;
• step iii);
• determining a difficulty level of the first corpus based on a similarity between the first corpus and the third corpus, wherein the difficulty level comprises at least two levels;
• step iv):
• determining the first corpus as a sample corpus in response to the difficulty level satisfying requirements of a difficulty level threshold, wherein the sample corpus is configured to optimize the to-be-optimized translation model;
• step v):
• acquiring a plurality of sample corpuses each being determined by steps i) to iv);
• step vi):
• sorting the plurality of sample corpuses from low to high according to difficulty levels of the sample corpuses to acquire a sample corpus sequence;
• and
• step vii):
• training the to-be-optimized translation model according to a sorting order indicated by the sample corpus sequence, wherein the training comprises:
• sequentially selecting, according to the sorting order, the sample corpuses as inputs of the to-be-optimized translation model.

A non-transitory computer readable storage medium, storing a computer instruction, wherein the computer instruction when executed by a computer causes the computer to perform operations comprising:

• step i):
• acquiring a first corpus, and translating the first corpus by using a to-be-optimized translation model to generate a second corpus corresponding to the first corpus, wherein a language type of the first corpus and a language type of the second corpus are different, and the to-be-optimized translation model is configured to perform bidirectional translation between the language type of the first corpus and the language type of the second corpus;
• step ii):
• translating the second corpus by using the to-be-optimized translation model to generate a third corpus corresponding to the second corpus, wherein a language type of the third corpus and the language type of the first corpus are the same;
• step iii);
• determining a difficulty level of the first corpus based on a similarity between the first corpus and the third corpus, wherein the difficulty level comprises at least two levels;
• step iv):
• determining the first corpus as a sample corpus in response to the difficulty level satisfying requirements of a difficulty level threshold, wherein the sample corpus is configured to optimize the to-be-optimized translation model;
• step v):
• acquiring a plurality of sample corpuses each being determined by steps i) to iv);
• step vi):
• sorting the plurality of sample corpuses from low to high according to difficulty levels of the sample corpuses to acquire a sample corpus sequence;
• and
• step vii):
• training the to-be-optimized translation model according to a sorting order indicated by the sample corpus sequence, wherein the training comprises:
• sequentially selecting, according to the sorting order, the sample corpuses as inputs of the to-be-optimized translation model.