Perturbation robust metric for evaluating image captions

A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

• receiving a training image and a ground truth image caption for the training image;
• generating a perturbed image caption for the training image by performing modifications to text elements of the ground truth image caption;
• generating, by a visual encoder, a visual embedding representation of the training image;
• generating, by a perturbation-aware text encoder, a first text embedding for the ground truth image caption and a second text embedding for the perturbed image caption;
• computing losses between the visual embedding representation of the training image, the first text embedding for the ground truth image caption, and the second text embedding for the perturbed image caption;
• and
• training the perturbation-aware text encoder based on the computed losses.

A computer-implemented method, comprising:

• receiving an image and an image caption, the image caption describing content in the image;
• generating, by a visual encoder, a visual embedding representation of the image;
• generating, by a perturbation-aware text encoder, a text embedding for the image caption, wherein the perturbation-aware text encoder is trained to recognize perturbations in image captions;
• and
• computing an image caption score for the image caption using the visual embedding representation of the image and text embedding for the image caption, wherein the image caption score is a metric indicating an accuracy of the image caption to the content in the image.