Using optical character recognition extraction and language model to populate an order with items from a recipe

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:

• receiving an image capturing a physical recipe;
• extracting a raw text from the received image;
• extracting a bounding box from the received image that includes one or more images of one or more ingredients of the physical recipe;
• generating a prompt for input into a generative model, the prompt including a task request for the generative model to generate a list of ingredients using the raw text and the bounding box including the one or more images;
• requesting the generative model to generate the list of ingredients using the prompt;
• formatting a text associated with the list of ingredients into a Hyper Text Markup Language (HTML) format by performing an ingredient tagging to generate a formatted list of ingredients;
• uploading the formatted list of ingredients into a cloud storage service;
• searching, using the formatted list of ingredients uploaded at the cloud storage service, a database of a computer system to find a list of items that match the list of ingredients;
• causing a device associated with a user of an online system to generate a user interface, wherein generating the user interface comprises rendering, in a single view, the list of items that is linked to a cart visually displayed at the user interface, the cart being updated with the list of items in response to a single selection of an add-all control element displayed at the user interface;
• constructing, using the formatted list of ingredients, a link for the list of ingredients;
• sending, through an application programming interface (API), the link to a web service of the online system;
• and
• responsive to an interaction with the link, causing the device associated with the user to update the user interface with the list of ingredients.

A computer system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium having instructions that, when executed by the processor, cause the computer system to perform steps comprising:
• receiving an image capturing a physical recipe;
• extracting a raw text from the received image;
• extracting a bounding box from the received image that includes one or more images of one or more ingredients of the physical recipe;
• generating a prompt for input into a generative model, the prompt including a task request for the generative model to generate a list of ingredients using the raw text and the bounding box including the one or more images;
• requesting the generative model to generate the list of ingredients using the prompt;
• formatting a text associated with the list of ingredients into a Hyper Text Markup Language (HTML) format by performing an ingredient tagging to generate a formatted list of ingredients;
• uploading the formatted list of ingredients into a cloud storage service;
• searching, using the formatted list of ingredients uploaded at the cloud storage service, a database of the computer system to find a list of items that match the list of ingredients;
• causing a device associated with a user of an online system to generate a user interface, wherein generating the user interface comprises rendering, in a single view, the list of items that is linked to a cart visually displayed at the user interface, the cart being updated with the list of items in response to a single selection of an add-all control element displayed at the user interface;
• constructing, using the formatted list of ingredients, a link for the list of ingredients;
• sending, through an application programming interface (API), the link to a web service of the online system;
• and
• responsive to an interaction with the link, causing the device associated with the user to update the user interface with the list of ingredients.