Application prediction based on a visual search determination

A computer-implemented method, the method comprising:

• obtaining, by a computing system comprising one or more processors and with an overlay application, display data, wherein the display data is descriptive of content currently presented for display on a user computing device;
• obtaining, by the computing system and with the overlay application, a gesture input from the user computing device;
• determining, by the computing system and with the overlay application, a region-of-interest within the dis lay data based on the gesture input and image features of the content currently presented for display;
• processing, by the computing system and with the overlay application, the display data with a segmentation model to generate a segmentation mask based on a detected object within the region-of-interest within the display data and to segment a portion of the display data based on the segmentation mask;
• processing, by the computing system, the portion of the display data to generate visual search data, wherein the visual search data comprises one or more visual search results, wherein the one or more visual search results are determined based on detected features within the portion of the display data;
• processing, by the computing system, the visual search data with a machine-learned suggestion model to determine a second application is associated with the one or more visual search results, wherein the second application is determined with the machine-learned suggestion model by:
• determining, by the computing system, one or more suggested actions based on the visual search data;
• and
• determining, by the computing system, the second application is associated with the one or more suggested actions;
• and
• providing, by the computing system and based on the visual search data, an application suggestion comprising an indication of the second application.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• obtaining, with an overlay application that is configured to obtain content provided for display from a plurality of different applications on a user computing device, display data, wherein the display data is descriptive of content presented for display on the user computing device;
• processing the display data with one or more on-device machine-learned models to generate a segmented portion of the display data, wherein the segmented portion of the display data comprises data descriptive of a set of features of the content presented for display, wherein processing the display data with the one or more on-device machine-learned models to generate the segmented portion of the display data comprises:
• determining, with the overlay application, a region-of-interest within the display data based on image features of the content currently presented for display;
• and
• processing, with the overlay application, the display data with a segmentation model to generate a segmentation mask based on a detected object within the region-of-interest within the display data and to generate the segmented portion of the display data based on the segmentation mask;
• transmitting the segmented portion of the display data to a server computing system;
• receiving visual search data from the server computing system, wherein the visual search data comprises one or more search results, wherein the one or more search results are determined based on detected features in the segmented portion of the display data;
• processing the visual search data with a machine-learned suggestion model to determine a plurality of candidate second applications that are associated with the one or more search results, wherein the plurality of candidate second applications are determined by:
• determining, with the machine-learned suggestion model, a plurality of suggested actions based on the visual search data;
• and
• determining, with the machine-learned suggestion model, the plurality of candidate second applications are associated with the one or more suggested actions;
• obtaining a selection of a particular application suggestion to transmit at least a portion of the visual search data to a particular second application of the plurality of candidate second applications;
• obtaining a model-generated content item based on the selection of the particular application suggestion, wherein the model-generated content item was generated with a generative model based on the portion of the visual search data;
• and
• providing the model-generated content item to the particular second application.