US 2025/0061146 A1Published application

Image query processing using large language models

AssigneeGoogle Inc.

InventorsOlivier Siegenthaler, Ágoston Weisz, Boris Bluntschli, Dan Banica, Kaan Ege Özgün, Daniel Mogoreanu, Filip Sladek

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Implementations utilize an LLM to respond to queries comprising image data, such as multimodal queries that include both text and image data. A natural language processing system is extended such that when an image is provided, the natural language processing system invokes one or more auxiliary image processing models (e.g., visual query) and/or image search engines. The results, of invoking such model(s) and/or search engine(s), are collected into structured data signals related to the image. These signals form part of the conversation context and are used to extend the text prompt that is sent to the LLM. This allows the LLM to take the context into account when being used to process the user query, thereby enabling generation of an LLM reply that addresses relevant feature(s) of the image.

Claims21

Dependent claims

A method implemented by one or more processors, the method comprising:

receiving an input query associated with a client device, the input query comprising an input image and an input text query, wherein the input text query refers to the input image and comprises one or more implicit queries;
generating, using an explication model and based on the input text query, one or more explicit text queries that explicate one or more of the implicit queries in the input text query;
processing, using a multi-modal image processing model, the input image and the one or more explicit text queries to generate one or more natural language descriptors, the one or more natural language descriptors descriptive of one or more properties of the input image, wherein the one or more natural language descriptors are responsive to the one or more explicit text queries;
generating, based on the one or more natural language descriptors and the input text query and/or the one or more explicit text queries, an input prompt for a large language model, LLM;
generating, from the input prompt and using the LLM, a response to the input query;
and
causing the response to the input query to be rendered at the client device.

4 more elements

A method implemented by one or more processors, the method comprising:

receiving an input query associated with a client device, the input query comprising an input image;
generating, based on the input image, an image search request for a search engine;
transmitting, to the search engine, the image search request;
receiving, from the search engine and in response to the image search request, a search response comprising one or more web resources containing at least one of one or more images responsive to the image search request;
extracting one or more text extracts from the one or more web resources;
generating, based on the one or more text extracts, an input prompt for a large language model, LLM;
generating, from the input prompt and using the LLM, a response to the input query;
and
causing the response to the input query to be rendered at a client device.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/802,734Filed 13 Aug 2024
Priority: 14 Aug 2023
Published: 20 Feb 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06F 16/3329
Query formulation: natural language query formulation
G06F 16/532
Querying: query formulation, e.g. graphical querying
G06F 40/40
Handling natural language data: processing or translation of natural language
G06V 10/803
Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level: of…