Open vocabulary food image recognition with multimodal generative models

A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to:

• receive one or more digital images depicting one or more food items;
• obtain one or more geolocations associated with one or more of the digital images;
• receive additional contextual data about one or more of the geolocations;
• assemble, as an input prompt for a generative model, data indicative of:
• one or more of the digital images depicting one or more food items, and
• the additional contextual data about the one or more geolocations;
• cause the input prompt to be processed using one or more generative models to generate one or more classifications of one or more of the food items, wherein the one or more classifications are conditioned on the additional contextual data;
• and
• cause one or more output devices to render output that conveys the one or more classifications.

At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to:

• receive one or more digital images depicting one or more food items;
• obtain one or more geolocations associated with one or more of the digital images;
• receive additional contextual data about one or more of the geolocations;
• assemble, as an input prompt for a generative model, data indicative of:
• one or more of the digital images depicting one or more food items, and
• the additional contextual data about the one or more geolocations;
• cause the input prompt to be processed using one or more generative models to generate one or more classifications of one or more of the food items, wherein the one or more classifications are conditioned on the additional contextual data;
• and
• cause one or more output devices to render output that conveys the one or more classifications.