Text-conditioned visual attention for multimodal machine learning models

A system comprising:

• one or more memory devices comprising a reference image, a target image, a modification text, and a machine learning model;
• and
• one or more processors configured to cause the system to:
• generate, utilizing the machine learning model, a reference text-image feature vector from the reference image and the modification text;
• generate, utilizing the machine learning model, a target text-image feature vector from the target image and the modification text;
• generate a multi-modal gradient attention map utilizing the reference text-image feature vector and the target text-image feature vector comprising the modification text by utilizing a gradient of a similarity metric between the reference text-image feature vector and the target text-image feature vector with respect to parameters of the machine learning model;
• and
• modify the parameters of the machine learning model by comparing the multi-modal gradient attention map and a saliency map.

A non-transitory computer-readable medium storing executable instructions that, when executed by a processing device, cause the processing device to perform operations comprising:

• generating, utilizing a vision-language neural network of a machine learning model, a reference text-image feature vector based on a reference image and a modification text;
• generating, utilizing the vision-language neural network of the machine learning model, a target text-image feature vector based on a target image and the modification text;
• determining, utilizing the machine learning model, a predicted measure of similarity between the reference image conditioned on the modification text and the target image;
• and
• generating, from the reference text-image feature vector and the target text-image feature vector comprising the modification text, a multi-modal gradient attention map by utilizing a gradient of the predicted measure of similarity between the reference text-image feature vector and the target text-image feature vector with respect to parameters of the vision-language neural network, the multi-modal gradient attention map reflecting a visual grounding of the machine learning model in determining the predicted measure of similarity.