Adaptive cycle consistency multimodal image captioning

A computer program product comprising:

• one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the stored program instructions comprising:
• program instructions to encode an image utilizing an image encoder, wherein the image is contained within a triplet comprising the image, one or more high-resource captions, and one or more low-resource captions;
• program instructions to generate one or more high-resource captions utilizing the encoded image and the triplet inputted into a high-resource decoder;
• program instructions to encode the one or more generated high-resource captions utilizing a high-resource encoder;
• program instructions to add adaptive cycle consistency constraints on a set of calculated attention weights associated the triplet;
• and
• program instructions to generate one or more low-resource captions by simultaneously inputting the encoded image, the encoded high-resource caption, and the triplet into a trained low-resource decoder.

A computer system comprising:

• one or more computer processors;
• one or more computer readable storage media;
• and
• program instructions stored on the computer readable storage media for execution by at least one of the one or more processors, the stored program instructions comprising:
• program instructions to encode an image utilizing an image encoder, wherein the image is contained within a triplet comprising the image, one or more high-resource captions, and one or more low-resource captions;
• program instructions to generate one or more high-resource captions utilizing the encoded image and the triplet inputted into a high-resource decoder;
• program instructions to encode the one or more generated high-resource captions utilizing a high-resource encoder;
• program instructions to add adaptive cycle consistency constraints on a set of calculated attention weights associated the triplet;
• and
• program instructions to generate one or more low-resource captions by simultaneously inputting the encoded image, the encoded high-resource caption, and the triplet into a trained low-resource decoder.