Multimodal translation method, apparatus, electronic device and computer-readable storage medium

An electronic device for multimodal translation of a content in a source language, comprising:

• at least one processor;
• and
• a memory configured to store instructions that, when executed, cause the at least one processor to:
• receive a user input with respect to a translation request of text included in the content;
• in response to receiving the user input, acquire a multimodal input from a content, the multimodal input including location information related to the content and other multimodal inputs;
• generate scene information representing the multimodal input related to the content by using a fusion layer based on the location information and the other multimodal inputs;
• identify a candidate word set in a target language;
• determine at least one candidate word from the candidate word set based on the scene information;
• and
• translate the text included in the content into the target language using a translation model based on the determined at least one candidate word.

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to:

• receive a user input with respect to a translation request of text included in a content in a source language;
• in response to receiving the user input, acquire a multimodal input from a content, the multimodal input including location information related to the content and other multimodal inputs;
• generate scene information representing the multimodal input related to the content by using a fusion layer based on the location information and the other multimodal inputs;
• identify a candidate word set in a target language;
• determine at least one candidate word from the candidate word set based on the scene information;
• and
• translate the text included in the content into the target language using a translation model based on the determined at least one candidate word.