# US 2026/0065351 A1 · Using optical character recognition extraction and language model to populate an order with items from a recipe

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065351 A1
- Document ID: US20260065351A1
- Status: Published application
- Kind: A1
- Application number: 19/382,105
- Filed: 2025-11-06
- Published: 2026-03-05
- Earliest claimed: 2023-08-29
- Assignees: Maplebear Inc.
- Applicants: Maplebear Inc.
- Inventors: Noah Finkielsztein; Weiyue Li; Muhammad Aun; Ilya Dyoshin
- Web page: https://patentgrep.com/patent/US20260065351A1

## CPC

- G06Q 30/06333: {virtual shopping carts}
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G06F 40/205: Parsing
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06Q 30/0617: {Representative agent}
- G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06V 30/1448: {based on markings or identifiers characterising the document or the area}
- G06V 30/30: based on the type of data
- G06V 30/18: Extraction of features or characteristics of the image
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments relate to utilizing an optical character recognition extraction and a large language model (LLM) to automatically populate a shopping cart of a user of an online system with items from a physical recipe. The online system receives an image capturing the physical recipe and extracts a raw text from the received image. The online system generates a prompt for input into the LLM, the prompt including a task request for the LLM to generate a list of ingredients using the raw text. The online system inputs the prompt into the LLM to generate the list of ingredients. The online system maps the list of ingredients to a list of items available by one or more retailers associated with the online system. The online system causes a device of the user to display a user interface with the list of items.

## Claims

### Claim 1

A method comprising, at a computer system comprising a processor and a computer-readable medium:
receiving an image capturing a physical recipe;
extracting a raw text from the received image;
generating a prompt for input into a generative model, the prompt including a task request for the generative model to generate a list of ingredients using the raw text;
requesting the generative model to generate the list of ingredients using the prompt;
formatting a text associated with the list of ingredients into a Hyper Text Markup Language (HTML) format to generate a formatted list of ingredients;
searching, using the formatted list of ingredients, a database of the computer system to find a list of items that match the list of ingredients;
causing a device associated with a user of an online system to generate a user interface, wherein generating the user interface comprises rendering the list of items that is linked to a cart visually displayed at the user interface;
constructing, using the formatted list of ingredients, a link for the list of ingredients;
sending, through an application programming interface (API), the link to a web service of the online system; and
responsive to an interaction with the link, causing the device associated with the user to update the user interface with the list of ingredients.

### Claim 2 (depends on 1)

The method of claim 1, wherein extracting the raw text comprises:
extracting the raw text from the received image using an optical character recognition (OCR) algorithm.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
parsing the extracted raw text to exclude a non-textual portion from the raw text prior to generating the prompt for input into the generative model.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating the prompt for input into the generative model further comprises:
including information into the prompt that the raw text relates to a recipe.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
extracting a bounding box from the received image that includes one or more images of one or more ingredients of the physical recipe; and
generating the prompt by further including, into the prompt, information about the bounding box including the one or more images.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
uploading the formatted list of ingredients into a cloud storage service of the computer system.

### Claim 7 (depends on 1)

The method of claim 1, wherein searching the database comprises:
searching the database using information about an item type for each ingredient in the formatted list of ingredients to identify a plurality of candidate items that match item types for the formatted list of ingredients; and
selecting the list of items from the plurality of candidate items.

### Claim 8 (depends on 7)

The method of claim 7, wherein selecting the list of items comprises:
scoring each candidate item of the plurality of candidate items; and
selecting the list of items based on a score of each candidate item of the plurality of candidate items.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
automatically adding the list of items to the cart.

### Claim 10 (depends on 1)

The method of claim 1, wherein causing the device to generate the user interface comprises:
generating the user interface to display a control element for each item in the list of items for unselecting each item from the list of items prior to including one or more items from the list of items into the cart.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
identifying a set of items associated with a set of ingredients from the formatted list of ingredients,
wherein causing the device to generate the user interface further comprises generating the user interface that includes the identified set of items for adding to the cart in addition to one or more items from the list of items.

### Claim 12

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
receiving an image capturing a physical recipe;
extracting a raw text from the received image;
generating a prompt for input into a generative model, the prompt including a task request for the generative model to generate a list of ingredients using the raw text;
requesting the generative model to generate the list of ingredients using the prompt;
formatting a text associated with the list of ingredients into a Hyper Text Markup Language (HTML) format to generate a formatted list of ingredients;
searching, using the formatted list of ingredients, a database of a computer system to find a list of items that match the list of ingredients;
causing a device associated with a user of an online system to generate a user interface, wherein generating the user interface comprises rendering the list of items that is linked to a cart visually displayed at the user interface;
constructing, using the formatted list of ingredients, a link for the list of ingredients;
sending, through an application programming interface (API), the link to a web service of the online system; and
responsive to an interaction with the link, causing the device associated with the user to update the user interface with the list of ingredients.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
extracting the raw text from the received image using an optical character recognition (OCR) algorithm.

### Claim 14 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
generating the prompt for input into the generative model by further including information into the prompt that the raw text relates to a recipe.

### Claim 15 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
extracting a bounding box from the received image that includes one or more images of one or more ingredients of the physical recipe; and
generating the prompt by further including, into the prompt, information about the bounding box including the one or more images.

### Claim 16 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
uploading the formatted list of ingredients into a cloud storage service of the computer system.

### Claim 17 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
searching the database using information about an item type for each ingredient in the formatted list of ingredients to identify a plurality of candidate items that match item types for the formatted list of ingredients;
scoring each candidate item of the plurality of candidate items; and
selecting the list of items based on a score of each candidate item of the plurality of candidate items.

### Claim 18 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
automatically adding the list of items to the cart.

### Claim 19 (depends on 12)

The computer program product of claim 12, wherein the instructions further cause the processor to perform steps comprising:
causing the device of the user to generate the user interface to display a control element for each item in the list of items for unselecting each item from the list of items prior to including one or more items from the list of items into the cart.

### Claim 20

A computer system comprising:
a processor; and
a non-transitory computer-readable storage medium having instructions that, when executed by the processor, cause the computer system to perform steps comprising:
receiving an image capturing a physical recipe;
extracting a raw text from the received image;
generating a prompt for input into a generative model, the prompt including a task request for the generative model to generate a list of ingredients using the raw text;
requesting the generative model to generate the list of ingredients using the prompt;
formatting a text associated with the list of ingredients into a Hyper Text Markup Language (HTML) format to generate a formatted list of ingredients;
searching, using the formatted list of ingredients, a database of the computer system to find a list of items that match the list of ingredients;
causing a device associated with a user of an online system to generate a user interface, wherein generating the user interface comprises rendering the list of items that is linked to a cart visually displayed at the user interface;
constructing, using the formatted list of ingredients, a link for the list of ingredients;
sending, through an application programming interface (API), the link to a web service of the online system; and
responsive to an interaction with the link, causing the device associated with the user to update the user interface with the list of ingredients.
