Image captioning utilizing semantic text modeling and adversarial learning

A system for generating captions reflecting digital image contents, comprising:

• one or more memories, comprising:
• an image encoder neural network;
• and
• a sentence encoder neural network;
• at least one computing device storing instructions thereon, that, when executed by the at least one computing device, cause the system to:
• train the sentence encoder neural network utilizing an adversarial classifier and a reconstruction loss by:
• generating, utilizing the sentence encoder neural network, a predicted feature vector of a training sentence in a semantic space;
• applying the adversarial classifier to the predicted feature vector to constrain the sentence encoder neural network to regions of the semantic space that can be decoded to viable sentences;
• and
• train the image encoder neural network and the sentence encoder neural network utilizing a semantic similarity constraint by:
• generating, utilizing the image encoder neural network, a predicted image feature vector in the semantic space from a training image, wherein the training image corresponds to a plurality of training captions;
• generating, utilizing the sentence encoder neural network, a plurality of predicted caption feature vectors in the semantic space based on the plurality of training captions;
• and
• applying the semantic similarity constraint by:
• comparing the plurality of predicted caption feature vectors based on the plurality of training captions with the predicted image feature vector within the semantic space to determine a semantic similarity loss representing a difference in semantic meaning between the plurality of predicted caption feature vectors and the predicted image feature vector;
• and
• modifying parameters of the image encoder neural network and the sentence encoder neural network based on the semantic similarity loss.

A computer-implemented method for training a digital model to generate captions reflecting digital image contents, comprising:

• training a sentence encoder neural network utilizing an adversarial classifier by:
• generating, utilizing the sentence encoder neural network, a predicted feature vector of a training sentence in a semantic space;
• and
• applying the adversarial classifier to the predicted feature vector to constrain the sentence encoder neural network to regions of the semantic space that can be decoded to viable sentences;
• and
• training an image encoder neural network and the sentence encoder neural network utilizing a semantic similarity constraint by:
• generating, utilizing the image encoder neural network, a predicted image feature vector in the semantic space from a training image, wherein the training image corresponds to a plurality of training captions;
• generating, utilizing the sentence encoder neural network, a plurality of predicted caption feature vectors in the semantic space based on the plurality of training captions;
• and
• applying the semantic similarity constraint by:
• comparing the plurality of predicted caption feature vectors based on the plurality of training captions with the predicted image feature vector within the semantic space to determine a semantic similarity loss representing a difference in semantic meaning between the plurality of predicted caption feature vectors and the predicted image feature vector;
• and
• modifying parameters of the image encoder neural network and the sentence encoder neural network based on the semantic similarity loss.