Multi-modal model training method, apparatus and device, and storage medium

A multi-modal machine translation model training device, comprising:

• a memory, configured to store a computer program;
• and
• a processor, configured to execute the computer program to:
• obtain a training sample set, the training sample set comprising a plurality of training sample pairs, each of the training sample pairs comprising a visual sample and a corresponding original text;
• and
• train a multi-modal machine translation model for a plurality of rounds by successively using each of the training sample pairs in the training sample set until reaching a set end condition, wherein
• for any one of the training sample pairs in the training sample set, each time the multi-modal machine translation model is trained by using the training sample by the following steps:
• obtaining an image feature of a target visual sample comprised in the training sample pair;
• determining whether back translation needs to be performed on a target original text comprised in the training sample pair;
• when back translation needs to be performed on the target original text comprised in the training sample pair, the original text is input into a first translation model of a back translation module, intermediate texts are selected from output texts of the first translation model, and the intermediate texts are input into a second translation model of the back translation module, a target back-translated text is randomly selected among bask-translated texts output from the second translation model, and obtaining a text feature of the target back-translated text by performing language feature encoding on the target back-translated text;
• and
• training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target back-translated text.

A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is configured to, when executed by a processor, cause the processor to:

• obtain a training sample set, the training sample set comprising a plurality of training sample pairs, each of the training sample pairs comprising a visual sample and a corresponding original text;
• and
• train a multi-modal machine translation model for a plurality of rounds by successively using each of the training sample pairs in the training sample set until reaching a set end condition, wherein
• for any one of the training sample pairs in the training sample set, each time the multi-modal machine translation model is trained by using the training sample by the following steps:
• obtaining an image feature of a target visual sample comprised in the training sample pair;
• determining whether back translation needs to be performed on a target original text comprised in the training sample pair;
• when back translation needs to be performed on the target original text comprised in the training sample pair, the original text is input into a first translation model of a back translation module, intermediate texts are selected from output texts of the first translation model, and the intermediate texts are input into a second translation model of the back translation module, a target back-translated text is randomly selected among back-translated texts output from the second translation model, and obtaining a text feature of the target back-translated text by performing language feature encoding on the target back-translated text;
• and
• training the multi-modal machine translation model based on the image feature of the target visual sample and the text feature of the target back-translated text.