# US 2021/0241354 A1 · Voice-enabled recipe selection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241354 A1
- Document ID: US20210241354A1
- Status: Published application
- Kind: A1
- Application number: 16/779,535
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Walmart Apollo, LLC
- Applicants: Walmart Apollo, LLC
- Inventors: Snehasish Mukherjee; Deepa Mohan; Haoxuan Chen; Phani Ram Sayapaneni; Ghodratollah Aalipour Hafshejani; Shankara Bhargava Subramanya
- Web page: https://patentgrep.com/patent/US20210241354A1

## CPC

- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/22: Indexing; Data structures therefor; Storage structures
- G06F 16/3329: Natural language query formulation
- G06F 40/30: Semantic analysis
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06Q 30/0603: {Catalogue creation or management}
- G06Q 30/0619: {Neutral agent}
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 2015/223: {Execution procedure of a spoken command}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system including one or more processors and one or more non-transitory computer-readable media storing computing instructions configured to run on the one or more processors and perform receiving a voice command from a user; transforming the voice command; transforming the voice command can include using a natural language understanding and rules execution engine into (a) an intent of the user to add recipe ingredients to a cart and (b) a recipe descriptor; determining a matching recipe from a set of ingested recipes based on the recipe descriptor; determining items and quantities associated with the items that correspond to a set of ingredients included in the matching recipe using a quantity inference algorithm; and automatically adding all of the items and the quantities associated with the items to the cart. Other embodiments are disclosed.

## Claims

### Claim 1

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computing instructions configured to run on the one or more processors and perform:
receiving a voice command from a user;
transforming the voice command, using a natural language understanding and rules execution engine, into (a) an intent of the user to add recipe ingredients to a cart and (b) a recipe descriptor;
upon detection of the intent of the user to add the recipe ingredients to the cart, determining a matching recipe from a set of ingested recipes based on the recipe descriptor;
determining items and quantities associated with the items that correspond to a set of ingredients included in the matching recipe using a quantity inference algorithm; and
automatically adding all of the items and the quantities associated with the items to the cart.

### Claim 2 (depends on 1)

The system of claim 1, wherein the computing instructions are further configured to perform, before receiving the voice command from the user:
pre-processing a plurality of recipes, wherein the pre-processing comprises, for each recipe of the plurality of recipes:
retrieving a respective textual description for the each recipe, wherein the respective textual description comprises a respective title of the each recipe and respective recipe ingredients of the each recipe;
extracting respective tuples from the respective textual description using a machine learning model, as trained, wherein the respective tuples represent the respective recipe ingredients for the each recipe;
generating a respective mapping from the recipe title to the respective tuples of the each recipe; and
storing the respective mapping for the each recipe in an ingested recipe database, wherein the set of ingested recipes comprises the plurality of recipes.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
the machine learning model includes a name entity recognition (NER) model with a Bi-LSTM CRF, wherein the Bi-LSTM CRF is a recurrent neural network comprising a combination of a long short-term memory (LSTM) and a conditional random field (CRF).

### Claim 4 (depends on 2)

The system of claim 2, wherein:
each of the respective tuples comprises an item, a size, a quantity, and a unit of a respective one of the respective recipe ingredients of the each recipe.

### Claim 5 (depends on 1)

The system of claim 1, wherein determining the items and the quantities further comprises:
determining the set of ingredients for the matching recipe based on respective tuples for the matching recipe.

### Claim 6 (depends on 5)

The system of claim 5, wherein determining the items and the quantities further comprises:
determining respective available items in an item catalog that correspond to each item stored in the respective tuples using a search engine; and
the quantity inference algorithm uses the respective available items to determine a respective closest quantity for the each item.

### Claim 7 (depends on 1)

The system of claim 1, wherein determining the matching recipe from the set of ingested recipes further comprises:
using a fuzzy matching algorithm with a damerau-levenshtein distance.

### Claim 8 (depends on 7)

The system of claim 7, wherein:
the fuzzy matching algorithm uses elastic search to index the set of ingested recipes; and
the fuzzy matching algorithm matches the matching recipe to the set of ingested recipes, as indexed.

### Claim 9 (depends on 1)

The system of claim 1, wherein determining the items and the quantities further comprises:
calculating a closest quantity of an item of the items for the matching recipe using the quantity inference algorithm based on one or more available items in an item catalog.

### Claim 10 (depends on 1)

The system of claim 1, wherein transforming the voice command further comprises:
using pattern templates in the natural language understanding and rules execution engine to decipher the intent of the user and the recipe descriptor from the voice command.

### Claim 11

A method being implemented via execution of computing instructions configured to run at one or more processors and stored at one or more non-transitory computer-readable media, the method comprising:
receiving a voice command from a user;
transforming the voice command, using a natural language understanding and rules execution engine, into (a) an intent of the user to add recipe ingredients to a cart and (b) a recipe descriptor;
upon detection of the intent of the user to add the recipe ingredients to the cart, determining a matching recipe from a set of ingested recipes based on the recipe descriptor;
determining items and quantities associated with the items that correspond to a set of ingredients included in the matching recipe using a quantity inference algorithm; and
automatically adding all of the items and the quantities associated with the items to the cart.

### Claim 12 (depends on 11)

The method of claim 11, further comprising, before receiving the voice command from the user:
pre-processing a plurality of recipes, wherein the pre-processing comprises, for each recipe of the plurality of recipes:
retrieving a respective textual description for the each recipe, wherein the respective textual description comprises a respective title of the each recipe and respective recipe ingredients of the each recipe;
extracting respective tuples from the respective textual description using a machine learning model, as trained, wherein the respective tuples represent the respective recipe ingredients for the each recipe;
generating a respective mapping from the recipe title to the respective tuples of the each recipe; and
storing the respective mapping for the each recipe in an ingested recipe database, wherein the set of ingested recipes comprises the plurality of recipes.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the machine learning model includes a name entity recognition (NER) model with a Bi-LSTM CRF, wherein the Bi-LSTM CRF is a recurrent neural network comprising a combination of a long short-term memory (LSTM) and a conditional random field (CRF).

### Claim 14 (depends on 12)

The method of claim 12, wherein:
each of the respective tuples comprises an item, a size, a quantity, and a unit of a respective one of the respective recipe ingredients of the each recipe.

### Claim 15 (depends on 11)

The method of claim 11, wherein:
determining the set of ingredients for the matching recipe based on respective tuples for the matching recipe.

### Claim 16 (depends on 15)

The method of claim 15, wherein determining the items and the quantities further comprises:
determining respective available items in an item catalog that correspond to each item stored in the respective tuples using a search engine; and
the quantity inference algorithm uses the respective available items to determine a respective closest quantity for the each item.

### Claim 17 (depends on 11)

The method of claim 11, wherein determining the matching recipe from the set of ingested recipes further comprises:
using a fuzzy matching algorithm with a damerau-levenshtein distance.

### Claim 18 (depends on 17)

The method of claim 17, wherein
the fuzzy matching algorithm uses elastic search to index the set of ingested recipes; and
the fuzzy matching algorithm matches the matching recipe to the set of ingested recipes, as indexed.

### Claim 19 (depends on 11)

The method of claim 11, wherein determining the items and the quantities further comprises:
calculating a closest quantity of an item of the items for the matching recipe using the quantity inference algorithm based on one or more available items in an item catalog.

### Claim 20 (depends on 11)

The method of claim 11, wherein transforming the voice command further comprises:
using pattern templates in the natural language understanding and rules execution engine to decipher the intent of the user and the recipe descriptor from the voice command.