Using a large language model to generate content based on images captured at a source location

A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:

• determining that stored information describing an item stocked at a source location is stale;
• responsive to determining that the stored information is stale, transmitting an instruction to a picker device to capture one or more images of the item at the source location along with text displayed at the source location describing the item;
• receiving, at a computing system, the one or more images captured at the source location from the picker device, the one or more images depicting one or more objects comprising the item and the text;
• generating a prompt comprising:
• the one or more images captured at the source location,
• a request to identify, from the one or more objects depicted in the one or more images, a set of items available at the source location based at least in part on a database of items available at the source location, and
• a request to extract, for each identified item, the text from the one or more images, wherein the text describes one or more of a price or a promotion;
• providing the prompt to a multi-modal large language model to obtain an output, wherein the multi-modal large language model is fine-tuned based at least in part on the database of items available at the source location;
• extracting, from the output of the multi-modal large language model, an identifier and the text associated with each item of the set of items available at the source location;
• retrieving a set of item data for each item of the set of items based at least in part on the identifier associated with a corresponding item;
• generating content for the source location based at least in part on the set of item data and the one or more of the price or the promotion associated with each item of the set of items;
• and
• sending the content for the source location to a client device associated with a user, causing the client device to display the content.

A computing system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, perform actions comprising:
• determining that stored information describing an item stocked at a source location is stale;
• responsive to determining that the stored information is stale, transmitting an instruction to a picker device to capture one or more images of the item at the source location along with text displayed at the source location describing the item;
• receiving, at the computing system, the one or more images captured at the source location from the picker device, the one or more images depicting one or more objects comprising the item and the text;
• generating a prompt comprising:
• the one or more images captured at the source location,
• a request to identify, from the one or more objects depicted in the one or more images, a set of items available at the source location based at least in part on a database of items available at the source location, and
• a request to extract, for each identified item, the text from the one or more images, wherein the text describes one or more of a price or a promotion;
• providing the prompt to a multi-modal large language model to obtain an output, wherein the multi-modal large language model is fine-tuned based at least in part on the database of items available at the source location;
• extracting, from the output of the multi-modal large language model, an identifier and the text associated with each item of the set of items available at the source location;
• retrieving a set of item data for each item of the set of items based at least in part on the identifier associated with a corresponding item;
• generating content for the source location based at least in part on the set of item data and the one or more of the price or the promotion associated with each item of the set of items;
• and
• sending the content for the source location to a client device associated with a user, causing the client device to display the content.