Using visual language models to determine locations of image elements within graphical images

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform operations for training a visual language model to identify locations of image elements within a graphical image, the operations comprising:

• generating a plurality of training data items, each training data item including (i) a graphical image rendered according to a corresponding set of instructions, (ii) a natural language query for identifying at least one image element of the graphical image, and (iii) a target location for the at least one image element, the target location being determined from the set of instructions;
• for each of the training data items, processing the corresponding graphical image and natural language query using a visual language model to generate a corresponding model output comprising a predicted location of an image element identified from the natural language query;
• and
• adjusting parameters of the visual language model to optimize, for each of the training data items, an objective function that depends on a comparison between the predicted location of the model output corresponding to the training data item and the target location of the training data item.

One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform operations for training a visual language model to identify locations of image elements within a graphical image, the operations comprising:

• generating a plurality of training data items, each training data item including (i) a graphical image rendered according to a corresponding set of instructions, (ii) a natural language query for identifying at least one image element of the graphical image, and (iii) a target location for the at least one image element, the target location being determined from the set of instructions;
• for each of the training data items, processing the corresponding graphical image and natural language query using a visual language model to generate a corresponding model output comprising a predicted location of an image element identified from the natural language query;
• and
• adjusting parameters of the visual language model to optimize, for each of the training data items, an objective function that depends on a comparison between the predicted location of the model output corresponding to the training data item and the target location of the training data item.