Cross-Modal Processing For Vision And Language

A computer-implemented method, comprising:

• extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein each visual feature is corresponding to a pixel block in the input image;
• determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary;
• extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature is corresponding to at least one word in the input text;
• and
• determining association information between the input text and the input image based on the set of visual semantic features and the set of text features.

An electronic device, comprising:

• a processing unit;
• and
• a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising:
• extracting a set of visual features of a training image according to a visual feature extraction sub-model in a target model, wherein each visual feature is corresponding to a pixel block in the training image;
• determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features;
• extracting a set of text features of a training text corresponding to the training image according to a text feature extraction sub-model in the target model, wherein each text feature is corresponding to at least one word in the training text;
• and
• training, based on the set of visual semantic features and the set of text features, the target model to determine association information between an input text and an input image.

An electronic device, comprising:

• a processing unit;
• and
• a memory coupled to the processing unit and comprising instructions stored thereon which, when executed by the processing unit, cause the device to perform acts comprising:
• extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein each visual feature is corresponding to a pixel block in the input image;
• determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary;
• extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature is corresponding to at least one word in the input text;
• and
• determining association information between the input text and the input image based on the set of visual semantic features and the set of text features.

A computer program product, comprising machine-executable instructions which, when executed by a device, cause the device to perform acts comprising:

• extracting a set of visual features of a training image according to a visual feature extraction sub-model in a target model, wherein each visual feature is corresponding to a pixel block in the training image;
• determining, based on a visual semantic dictionary, a set of visual semantic features corresponding to the set of visual features;
• extracting a set of text features of a training text corresponding to the training image according to a text feature extraction sub-model in the target model, wherein each text feature is corresponding to at least one word in the training text;
• and
• training, based on the set of visual semantic features and the set of text features, the target model to determine association information between an input text and an input image.

A computer program product, comprising machine-executable instructions which, when executed by a device, cause the device to perform acts comprising:

• extracting a set of visual features of an input image according to a visual feature extraction sub-model in a trained target model, wherein each visual feature is corresponding to a pixel block in the input image;
• determining a set of visual semantic features corresponding to the set of visual features based on a visual semantic dictionary;
• extracting a set of text features of an input text corresponding to the input image according to a text feature extraction sub-model in the target model, wherein each text feature is corresponding to at least one word in the input text;
• and
• determining association information between the input text and the input image based on the set of visual semantic features and the set of text features.