Intent Determination-Based Visual Search Response Generation

A computer-implemented method for visual query processing, the method comprising:

• obtaining, by a computing system comprising one or more processors, an image query, wherein the image query comprises one or more images depicting a plurality of objects;
• processing, by the computing system, the image query with an object recognition model to generate a plurality of object labels, wherein the plurality of object labels are descriptive recognition labels associated with the plurality of objects;
• processing, by the computing system, the image query and the plurality of object labels with a semantic understanding model to generate a semantic understanding output, wherein the semantic understanding output is descriptive of relationships of the plurality of objects within the one or more images;
• determining, by the computing system and via a search engine, a plurality of content items associated with the image query, wherein the plurality of content items are determined based on identifying a plurality of visual matches associated with the image query;
• and
• processing, by the computing system, the plurality of object labels, semantic understanding output, and at least a subset of the plurality of content items with a generative language model to generate a model-generated response to the image query, and wherein the model-generated response comprises details determined based on the at least a subset of the plurality of content items.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• obtaining an image query via a viewfinder interface of a search application, wherein the image query comprises one or more images depicting a plurality of features;
• processing the image query with a machine-learned model to generate an intent determination, wherein the machine-learned model was trained to process images and determine a query intent based on the images, wherein the intent determination is descriptive of a predicted search intent of a user that input the one or more images, wherein the predicted search intent is determined based on identifying objects within the one or more images and determining an image classification;
• determining a particular prompt is associated with the intent determination, wherein the particular prompt is obtained from a prompt library comprising a plurality of different prompts;
• determining a plurality of content items associated with the image query, wherein the plurality of content items are determined based on identifying a plurality of visual matches associated with the image query;
• processing the particular prompt and at least a subset of the plurality of content items with a generative language model to generate a model-generated response to the image query, wherein the model-generated response comprises a format based on the intent determination, and wherein the model-generated response comprises details determined based on the at least a subset of the plurality of content items;
• and
• providing the model-generated response and a plurality of search results for display within a search results interface of the search application.