# US 11,907,901 B2 · Systems and methods of product recognition through multi-model image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,901 B2
- Document ID: US11907901B2
- Status: Granted
- Kind: B2
- Application number: 18/101,902
- Filed: 2023-01-26
- Published: 2023-06-08
- Granted: 2024-02-20
- Earliest claimed: 2019-02-25
- Assignees: Walmart Apollo, LLC
- Applicants: Walmart Apollo, LLC
- Inventors: Michael A. Garner; Priyanka Paliwal
- Web page: https://patentgrep.com/patent/US11907901B2

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06Q 10/08741: {by picking of items from inventory for fulfillment}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06Q 20/12: specially adapted for electronic shopping systems
- G06Q 20/201: {Price look-up processing, e.g. updating}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 20/3221: {Access to banking information through M-devices}
- G06Q 20/3223: {Realising banking transactions through M-devices}
- G06Q 20/326: {Payment applications installed on the mobile devices}
- G06Q 20/42: Confirmation, e.g. check or permission by the legal debtor of payment
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/776: Validation; Performance evaluation
- G06V 20/20: in augmented reality scenes
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration}
- G06N 3/08: Learning methods
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

In some embodiments, systems and methods are provided to recognize retail products, comprising: a model training system configured to: identify a customer; access an associated customer profile; access and apply a set of filtering rules to a product database based on customer data; generate a listing of products specific to the customer; access and apply a model training set of rules to train a machine learning model based on the listing of products and corresponding image data for each of the products in the listing of products; and communicate the trained model to the portable user device associated with first customer.

## Claims

### Claim 1

A system to recognize retail products, comprising:
an imaging system configured to capture a series of images;
at least one tangible memory storing a local product database comprising sets of product imaging data, wherein each set of product imaging data corresponds to one of multiple different retail products; and
a decision control circuit communicatively coupled with the memory and configured to:
process each image of a set of images, captured by the imaging system at different orientations relative to an item within each image of the set of images, by at least a first modeling technique relative to a first image attribute and obtain a corresponding first product identification probability that the item, captured within each of the set of images, is estimated to be a first product;
process each image of the set of images by a second modeling technique relative to a second image attribute that is different than the first image attribute, and obtain a corresponding second product identification probability that the item, captured within each of the set of images, is estimated to be the first product, and providing first product identification probabilities each corresponding to one of the images of the set of images, and second product identification probabilities each corresponding to one of the images of the set of images;
collectively evaluate the first product identification probabilities and the second product identification probabilities; and
identify the item as the first product in response to identifying a predefined relationship with a collective threshold probability based on the collective evaluation of the first product identification probabilities and the second product identification probabilities has a predefined relationship with a collective threshold probability.

### Claim 2 (depends on 1)

The system of claim 1, wherein the decision control circuit is configured to determine an aggregated first identification probability of the first product as a function of the first product identification probabilities corresponding to the set of images, and wherein the decision control circuit, in identifying the predefined relationship with the collective threshold probability, is configured to identify that at least the aggregated first identification probability has the predefined relationship with the collective threshold probability.

### Claim 3 (depends on 1)

The system of claim 1, wherein the decision control circuit is configured to:
determine an aggregated first identification probability of the first product as a function of the first product identification probabilities corresponding to the set of images;
determine an aggregated second identification probability of the first product as a function of the second product identification probabilities corresponding to the set of images, and
wherein the decision control circuit in, identifying the predefined relationship with the collective threshold probability, is configured to identify the aggregated first identification probability and the aggregated second identification probability have the predefined relationship with the collective threshold probability.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first modeling technique comprises optical character recognition (OCR) modeling technique and the decision control circuit in processing each image of the set of images according to the first modeling technique is configured to process each image of the set of images by the OCR modeling technique relative to text image attributes, and obtain corresponding text product identification probabilities that the item, captured within each image of the set of images, is estimated to be the first product; and
wherein the second modeling technique comprises a barcode recognition modeling technique and the second image attribute comprises a barcode image attribute, wherein the decision control circuit in obtaining the second product identification probabilities is configured to process each image of the set of images by the barcode recognition modeling technique relative to the barcode image attribute, and obtain corresponding barcode product identification probabilities that the item, captured within each image of the set of images, is estimated to be the first product.

### Claim 5 (depends on 1)

The system of claim 1, wherein the decision control circuit in collectively evaluating the first product identification probabilities and the second product identification probabilities is configured to:
apply a first weighting relative to the first product identification probabilities as a function of an expected degree of accuracy relative to the first image attribute to provide at least one weighted first product identification probability;
apply a second weighting relative to the second product identification probabilities as a function of an expected degree of accuracy relative to the second image attribute to provide at least one weighted second product identification probability; and
identify when there is a threshold consistency between the at least one weighted first product identification probability and the at least one weighted second product identification probability.

### Claim 6 (depends on 5)

The system of claim 5, wherein the decision control circuit is configured to identify a threshold inconsistency between an aggregated first identification probability, determined from the first product identification probabilities, and an aggregated second identification probability, determined from the second product identification probabilities, and apply at least one of the first weighting and the second weighting in response to identifying the threshold inconsistency between the aggregated first product identification probability and the aggregated second product identification probability.

### Claim 7 (depends on 5)

The system of claim 5, wherein the decision control circuit, in applying the first weighting to the first product identification probability, is configured to identify a number of textual words detected in each image of the set of images that are present on the image of the first product and multiply the number of textual words by a word multiplier to define the first weighting.

### Claim 8 (depends on 1)

The system of claim 1, wherein the decision control circuit is further configured to:
statistically process the first product identification probabilities and obtain an aggregated first identification probability;
statistically process the second product identification probabilities and obtain an aggregated second identification probability;
identify when there is a threshold inconsistency between the aggregated first identification probability and the aggregated second identification probability; and
apply, in response to determining there is a threshold inconsistency between the aggregated first identification probability and the aggregated second identification probability, a first weighting to the aggregated first identification probability as a function of an expected degree of accuracy relative to the first image attribute to provide a weighted first product identification probability, and determine a resultant identification probability of the item as the first product based on at least the weighted first product identification probability.

### Claim 9 (depends on 1)

The system of claim 1, further comprising:
an application configured to be executed by a control circuit of a portable user device wherein when executing the application the control circuit is configured to:
add the first product to a virtual cart based on the identification of the item as the first product; and
initiate a checkout of the virtual cart and receive payment for each product within the virtual shopping cart.

### Claim 10

A method to recognize retail products, comprising:
receiving a series of images;
processing each image of a set of images, captured by an imaging system at different orientations relative to an item within each image of the set of images by at least a first modeling technique relative to a first image attribute and obtaining a corresponding first product identification probability that the item, captured within each of the set of images, is estimated to be a first product;
processing each image of the set of images by a second modeling technique relative to a second image attribute that is different than the first image attribute and obtaining corresponding a second product identification probability that the item, captured within each of the set of images, is estimated to be the first product, and providing first product identification probabilities each corresponding to one of the images of the set of images, and second product identification probabilities each corresponding to one of the images of the set of images;
collectively evaluating the first product identification probabilities and the second product identification probabilities; and
identifying the item as the first product in response to identifying a predefined relationship with a collective threshold probability based on the collective evaluation of the first product identification probabilities and the second product identification probabilities.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
determining an aggregated first identification probability of the first product as a function of the first product identification probabilities corresponding to the set of images; and
wherein the identifying the predefined relationship with the collective threshold probability comprises identifying that the aggregated first identification probability has the predefined relationship with the collective threshold probability.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
determining an aggregated first identification probability of the first product as a function of the first product identification probabilities corresponding to the set of images;
determining an aggregated second identification probability of the first product as a function of the second product identification probabilities corresponding to the set of images; and
wherein the identifying the predefined relationship with the collective threshold probability comprises identifying that the aggregated first identification probability and the aggregated second identification probability have the predefined relationship with the collective threshold probability.

### Claim 13 (depends on 10)

The method of claim 10, wherein the processing each image of the set of images by the first modeling technique comprises processing each image of the set of images by an optical character recognition (OCR) modeling technique relative to text image attributes;
obtaining corresponding text product identification probabilities that the item, captured within each image of the set of images, is estimated to be the first product;
wherein the processing each image of the set of images by the second modeling technique comprises processing each image of the set of images by a barcode recognition modeling technique relative to a barcode image attribute; and
obtaining corresponding barcode product identification probabilities that the item, captured within each image of the set of images, is estimated to be the first product.

### Claim 14 (depends on 10)

The method of claim 10, wherein the collectively evaluating the first product identification probabilities and the second product identification probabilities comprises:
applying a first weighting relative to the first identification probabilities as a function of an expected degree of accuracy relative to the first image attribute to provide at least one weighted first product identification probability;
applying a second weighting relative to the second identification probabilities as a function of an expected degree of accuracy relative to the second image attribute to provide at least one weighted second product identification probability; and
identify when there is a threshold consistency between the at least one weighted first product identification probability and the at least one weighted second product identification probability.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
identifying a threshold inconsistency between an aggregated first identification probability, determined from the first product identification probabilities, and an aggregated second identification probability, determined from the second product identification probabilities; and
applying at least one of the first weighting and the second weighting in response to identifying the threshold inconsistency between the aggregated first product identification probability and the aggregated second product identification probability.

### Claim 16 (depends on 14)

The method of claim 14, wherein the applying the first weighting to the first product identification probability comprises identifying a number of textual words detected in each image of the set of images that are present on the image of the item; and
multiplying the number of textual words by a word multiplier to define the first weighting.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
statistically processing the first product identification probabilities and obtaining an aggregated first identification probability;
statistically processing the second product identification probabilities and obtaining an aggregated second identification probability;
identifying when there is a threshold inconsistency between the aggregated first identification probability and the aggregated second identification probability;
applying, in response to determining there is a threshold inconsistency between the aggregated first identification probability and the aggregated second identification probability, a first weighting to the aggregated first identification probability as a function of an expected degree of accuracy relative to the first image attribute to provide a weighted first product identification probability; and
determining a resultant identification probability of the item as the first product based on the weighted first product identification probability.

### Claim 18 (depends on 10)

The method of claim 10, further comprising:
adding the first product to a virtual cart; and
initiating a checkout of the virtual cart and receiving payment for each product within the virtual shopping cart.