Back to results
US 12,216,703 B2Granted
Visual search determination for text-to-image replacement
AssigneeGoogle Inc.
InventorsHarshit Kharbanda, Christopher James Kelley, Pendar Yousefi
Life of this patent
Estimating the term…
Abstract
Systems and methods for textual replacement can include the determination of a visual intent, which can trigger an interface for selecting an image to replace visual descriptors. The visually descriptive terms can be identified, and an indicator can be provided to indicate the text replacement option may be initiated. An image can then be selected by a user to replace the visually descriptive terms.
Claims20
A computer-implemented method for multimodal searching, the method comprising:
- obtaining, by a computing system comprising one or more processors, a search query, wherein the search query comprises one or more words and one or more additional words, wherein the one or more words comprise one or more visually descriptive terms, and wherein the one or more additional words are associated with a different descriptive aspect of the search query than the one or more words;
- processing, by the computing system, the search query with a machine-learned model to determine the one or more words comprise a visual intent, wherein the visual intent is associated with one or more visual features;
- in response to determining the one or more words comprise a visual intent, providing, by the computing system, an image-selection interface for display, wherein the image-selection interface comprises a plurality of images for selection, wherein the image-selection interface is provided for display based on the determination of the one or more words comprising the visual intent;
- obtaining, by the computing system, selection data, wherein the selection data is descriptive of a selection of an image;
- replacing, by the computing system, the one or more words with the image;
- providing, by the computing system, the image for display as replacement for the one or more words;
- determining, by the computing system, one or more search results associated with the one or more additional words and the one or more visual features of the image;
- and
- providing, by the computing system, the one or more search results as an output.
6 more elements
A computing system for text-to-image replacement, the system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- obtaining text data, wherein the text data is descriptive of a plurality of text characters, wherein the plurality of text characters are associated with a text string comprising one or more words and one or more additional words, wherein the one or more words comprise one or more visually descriptive terms, and wherein the one or more additional words are associated with a different descriptive aspect of the search query than the one or more words;
- processing the text data with a machine-learned model to determine a subset of the plurality of text characters associated with the one or more words comprise a visually-descriptive term, wherein the visually-descriptive term is associated with one or more visual features;
- in response to determining the one or more words comprise a visual intent, providing an image-selection interface for display, wherein the image-selection interface comprises a plurality of images for selection, wherein the plurality of images are obtained based at least in part on the visually-descriptive term;
- obtaining selection data, wherein the selection data is descriptive of a selection of an image;
- replacing the one or more words with the image to generate a multimodal search query;
- providing the image for display as replacement for the subset of the plurality of text characters;
- determining one or more search results associated with the one or more additional words and the one or more visual features of the image;
- and
- providing the one or more search results as an output.
9 more elements
One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
- obtaining a plurality of words, wherein the plurality of words comprise one or more particular words and one or more additional words, wherein the one or more particular words comprise one or more visually descriptive terms, and wherein the one or more additional words are associated with a different descriptive aspect of the search query than the one or more particular words;
- processing the plurality of words with a machine-learned model to determine the one or more particular words of the plurality of words comprise a visual intent, wherein the visual intent is associated with one or more visual features;
- in response to determining the one or more particular words comprise a visual intent, providing the plurality of words for display with an indicator identifying the one or more particular words;
- determining a plurality of images associated with the one or more particular words, wherein the plurality of images are associated with the visual intent;
- providing the plurality of images in a user interface panel, wherein the user interface panel comprises a plurality of interactive user interface elements associated with the plurality of images;
- obtaining a selection of a particular image of the plurality of images;
- replacing the one or more particular words with the particular image of the plurality of images to generate a multimodal search query;
- providing the multimodal search query for display, wherein the multimodal search query comprises the one or more additional words and the particular image without the one or more particular words,
- processing the multimodal search query to determine one or more search results associated with the one or more additional words and the one or more visual features of the image;
- and
- providing the one or more search results for display.
8 more elements
Dependencies are inferred from the claim text.