Enhancing vision language model understanding via visual search service-derived annotations

A computing system for training a vision language model to process annotated queries, the system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store computer-executable instructions for performing operations, the operations comprising:
• obtaining a training input comprising one or more images;
• processing the one or more images with a visual search engine to identify, by the visual search engine, one or more context items based on the one or more images, wherein the one or more context items comprise bounding shapes for one or more objects depicted in the one or more images;
• generating one or more annotations based on the one or more context items;
• annotating the training input with the one or more annotations to generate an annotated training input, wherein annotating the training input with the one or more annotations comprises overlaying the one or more bounding shapes upon at least one of the one or more images and adding reference numbers to the one or more bounding shapes and modifying the training input to contain textual references for the reference numbers;
• processing the annotated training input with a vision language model to generate, as an output of the vision language model, a response to the annotated training input;
• and
• modifying one or more values of one or more parameters of the vision language model based on a loss function that evaluates the response generated by the vision language model.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• receiving a query comprising one or more query images;
• processing the one or more query images with a visual search engine to identify, by the visual search engine, one or more context items based on the one or more query images, wherein the one or more context items comprise bounding shapes for one or more objects depicted in the one or more query images;
• generating one or more annotations based on the one or more context items;
• annotating the query with the one or more annotations to generate an annotated query, wherein annotating the query with the one or more annotations comprises overlaying the one or more bounding shapes upon at least one of the one or more query images and adding reference numbers to the one or more bounding shapes and modifying the query to contain textual references for the reference numbers;
• and
• processing the annotated query with a machine-learned vision language model to generate, as an output of the machine-learned vision language model, a response to the annotated query.