Method of processing multimedia data, device and medium

An electronic device, comprising:

• at least one processor;
• and
• a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, configured to cause the at least one processor to at least:
• recognize multimedia data, wherein a plurality of alternative multimedia data exist and the multimedia data comprises data of at least one selected from:
• an image modality, a text modality, or an audio modality, so as to obtain at least one key information of the multimedia data;
• query a predetermined knowledge base according to the at least one key information, so as to determine a multimedia name associated with the at least one key information and an association degree between the multimedia name and the at least one key information;
• and
• determine, in the multimedia name, a name of the multimedia data based on a similarity between alternative multimedia data for the multimedia name and the multimedia data, in response to the association degree being less than a first threshold value,
• wherein the instructions are further configured to cause the at least one processor to at least:
• encode the multimedia data so as to obtain first feature data of the multimedia data;
• determine a similarity between second feature data of each of the plurality of alternative multimedia data and the first feature data;
• and
• determine a multimedia name for target data as the name of the multimedia data, in response to the target data existing in the plurality of alternative multimedia data, wherein a similarity between second feature data of the target data and the first feature data is greater than or equal to a second threshold value;
• wherein the instructions are further configured to cause the at least one processor to at least:
• encode the data of the at least one modality, so as to obtain at least one feature sequence corresponding to the at least one modality respectively;
• fuse features contained in each feature sequence of the at least one feature sequence, so as to obtain a target feature for each feature sequence;
• and
• determine the first feature data according to the target feature for each feature sequence;
• and
• wherein the instructions are further configured to cause the at least one processor to at least:
• encode data of each modality of the at least one modality by using an encoder matched with the modality, so as to obtain a first sub feature sequence;
• encode the data of each modality by using a position encoder, so as to obtain a second sub feature sequence;
• randomly assign a modal feature to the data of each modality, so as to obtain a third sub feature sequence;
• and
• concatenate the first sub feature sequence, the second sub feature sequence and the third sub feature sequence, so as to obtain a feature sequence corresponding to each modality, wherein the modal feature is configured to represent each modality.

A non-transitory computer-readable storage medium having computer instructions therein, wherein the computer instructions are configured to cause a computer system to at least:

• recognize multimedia data, wherein a plurality of alternative multimedia data exist and wherein the multimedia data comprises data of at least one selected from:
• an image modality, a text modality, or an audio modality, so as to obtain at least one key information of the multimedia data;
• query a predetermined knowledge base according to the at least one key information, so as to determine a multimedia name associated with the at least one key information and an association degree between the multimedia name and the at least one key information;
• and
• determine, in the multimedia name, a name of the multimedia data based on a similarity between alternative multimedia data for the multimedia name and the multimedia data, in response to the association degree being less than a first threshold value,
• wherein the instructions configured to cause the computer system to determine the name of the multimedia are further configured to cause the at least computer system to at least:
• encode the multimedia data so as to obtain first feature data of the multimedia data;
• determine a similarity between second feature data of each of the plurality of alternative multimedia data and the first feature data;
• and
• determine a multimedia name for target data as the name of the multimedia data, in response to the target data existing in the plurality of alternative multimedia data, wherein a similarity between second feature data of the target data and the first feature data is greater than or equal to a second threshold value;
• wherein the instructions configured to cause the computer system to encode the multimedia data so as to obtain first feature data of the multimedia data are further configured to cause the computer system to at least:
• encode the data of the at least one modality, so as to obtain at least one feature sequence corresponding to the at least one modality respectively;
• fuse features contained in each feature sequence of the at least one feature sequence, so as to obtain a target feature for each feature sequence;
• and
• determine the first feature data according to the target feature for each feature sequence;
• and
• wherein the instructions configured to cause the computer system to encode the data of the at least one modality are further configured to cause the computer system to at least:
• encode data of each modality of the at least one modality by using an encoder matched with the modality, so as to obtain a first sub feature sequence;
• encode the data of each modality by using a position encoder, so as to obtain a second sub feature sequence;
• randomly assign a modal feature to the data of each modality, so as to obtain a third sub feature sequence;
• and
• concatenate the first sub feature sequence, the second sub feature sequence and the third sub feature sequence, so as to obtain a feature sequence corresponding to each modality, wherein the modal feature is configured to represent each modality.