# US 2026/0087833 A1 · Open vocabulary food image recognition with multimodal generative models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087833 A1
- Document ID: US20260087833A1
- Status: Published application
- Kind: A1
- Application number: 18/897,835
- Filed: 2024-09-26
- Published: 2026-03-26
- Earliest claimed: 2024-09-26
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Eric Cuiffo; Bo Lin
- Web page: https://patentgrep.com/patent/US20260087833A1

## CPC

- G06V 20/68: Food, e.g. fruit or vegetables
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 20/20: in augmented reality scenes
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 2201/10: Recognition assisted with metadata
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Implementations are described herein for improving the identification of entities in image data. In various implementations, image data is received depicting one or more food items. A geolocation associated with the image data can be obtained, as well as additional contextual data about one or more of the digital images. The image data along with the additional contextual data can be assembled into an input prompt for a generative model and processed by the generative model. The output of the generative mode can include a classification of one or more of the food items present in the image data. This classification can be rendered as output at a user device.

## Claims

### Claim 1

A method implemented using one or more processors, comprising:
receiving one or more digital images depicting one or more food items;
obtaining one or more geolocations associated with one or more of the digital images;
receiving additional contextual data about one or more of the geolocations;
assembling, as an input prompt for a generative model, data indicative of:
one or more of the digital images depicting one or more food items, and
the additional contextual data about the one or more geolocations;
causing the input prompt to be processed using one or more generative models to generate one or more classifications of one or more of the food items, wherein the one or more classifications are conditioned on the additional contextual data; and
causing one or more output devices to render output that conveys the one or more classifications.

### Claim 2 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises a name of a restaurant at or near one or more of the geolocations.

### Claim 3 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises a restaurant class or cuisine of a restaurant at or near one or more of the geolocations.

### Claim 4 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises one or more statements extracted from one or more user reviews of a restaurant at or near one or more of the geolocations.

### Claim 5 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises one or more price indicators associated with a restaurant at or near one or more of the geolocations.

### Claim 6 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises one or more menus of one or more restaurants corresponding to one or more of the geolocations.

### Claim 7 (depends on 6)

The method of claim 6, wherein a plurality of menu items contained in one or more of the menus define an ad hoc vocabulary of candidate dishes to which the one or more classifications, generated using the one or more generative models, are conditioned.

### Claim 8 (depends on 6)

The method of claim 6, wherein a plurality of menu items contained in one or more of the menus constrain a search space to which the one or more classifications, generated using the one or more generative models, are limited.

### Claim 9 (depends on 6)

The method of claim 6, wherein the input prompt is further assembled to include a command to match one or more of the food items depicted in one or more of the images with one or more menu items on one or more of the menus.

### Claim 10 (depends on 1)

The method of claim 1, wherein the additional contextual data comprises one or more documents one or more local dishes of a geographic region corresponding to one or more of the geolocations.

### Claim 11 (depends on 1)

The method of claim 1, wherein one or more of the generative models comprises a vision language model (VLM).

### Claim 12 (depends on 1)

The method of claim 1, wherein one or more of the generative models comprises a student model that is trained using a teacher model, wherein the student model has fewer parameters than the teacher model.

### Claim 13 (depends on 1)

The method of claim 1, wherein one or more of the geolocations comprises a geotag of one or more of the digital images.

### Claim 14 (depends on 1)

The method of claim 1, wherein one or more of the geolocations comprises position coordinates obtained by a mobile device carried by a user.

### Claim 15

A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:
receive one or more digital images depicting one or more food items;
obtain one or more geolocations associated with one or more of the digital images;
receive additional contextual data about one or more of the geolocations;
assemble, as an input prompt for a generative model, data indicative of:
one or more of the digital images depicting one or more food items, and
the additional contextual data about the one or more geolocations;
cause the input prompt to be processed using one or more generative models to generate one or more classifications of one or more of the food items, wherein the one or more classifications are conditioned on the additional contextual data; and
cause one or more output devices to render output that conveys the one or more classifications.

### Claim 16 (depends on 15)

The system of claim 15, wherein the additional contextual data comprises a name of a restaurant at or near one or more of the geolocations.

### Claim 17 (depends on 15)

The system of claim 15, wherein the additional contextual data comprises a restaurant class or cuisine of a restaurant at or near one or more of the geolocations.

### Claim 18 (depends on 15)

The system of claim 15, wherein the additional contextual data comprises one or more statements extracted from one or more user reviews of a restaurant at or near one or more of the geolocations.

### Claim 19 (depends on 15)

The system of claim 15, wherein the additional contextual data comprises one or more price indicators associated with a restaurant at or near one or more of the geolocations.

### Claim 20

At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to:
receive one or more digital images depicting one or more food items;
obtain one or more geolocations associated with one or more of the digital images;
receive additional contextual data about one or more of the geolocations;
assemble, as an input prompt for a generative model, data indicative of:
one or more of the digital images depicting one or more food items, and
the additional contextual data about the one or more geolocations;
cause the input prompt to be processed using one or more generative models to generate one or more classifications of one or more of the food items, wherein the one or more classifications are conditioned on the additional contextual data; and
cause one or more output devices to render output that conveys the one or more classifications.
