# US 12,118,509 B2 · Systems and methods of product recognition through multi-model image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,509 B2
- Document ID: US12118509B2
- Status: Granted
- Kind: B2
- Application number: 18/197,396
- Filed: 2023-05-15
- Published: 2023-09-07
- Granted: 2024-10-15
- Earliest claimed: 2019-02-25
- Assignees: Walmart Apollo, LLC
- Applicants: Walmart Apollo, LLC
- Inventors: Michael A. Garner
- Web page: https://patentgrep.com/patent/US12118509B2

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06F 16/535: Filtering based on additional data, e.g. user or group profiles
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/256: {of results relating to different input data, e.g. multimodal recognition}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06Q 10/08741: {by picking of items from inventory for fulfillment}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06Q 20/12: specially adapted for electronic shopping systems
- G06Q 20/201: {Price look-up processing, e.g. updating}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 20/3221: {Access to banking information through M-devices}
- G06Q 20/3223: {Realising banking transactions through M-devices}
- G06Q 20/326: {Payment applications installed on the mobile devices}
- G06Q 20/42: Confirmation, e.g. check or permission by the legal debtor of payment
- G06Q 30/0623: {by investigating goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/776: Validation; Performance evaluation
- G06V 20/20: in augmented reality scenes
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 20/62: Text, e.g. of license plates, overlay texts or captions on TV images
- G07G 1/0063: {with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration}
- G06N 3/08: Learning methods
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

In some embodiments, systems and methods are provided to recognize products in a physical facility through a portable device that comprises a decision control circuit configured to: process each frame of the subset of frames by multiple modeling techniques each relative to a corresponding image attribute and obtain a corresponding product identification probability; determine corresponding aggregated identification probabilities of the first product based on the product identification probabilities; collectively evaluate the aggregated identification probabilities and identify when a predefined relationship with a collective threshold probability exists; and cause an image of the first product to be displayed in response to identifying that one or more of the aggregated identification probabilities having the predefined relationship with the collective threshold probability.

## Claims

### Claim 1

A system to recognize products in a physical facility, comprising:
an imaging system;
a control circuit configured to control the imaging system to capture a series of frames;
at least one tangible memory storing a product database comprising sets of product imaging data, wherein each set of product imaging data corresponds to one of multiple different products; and
a decision control circuit communicatively coupled with the memory and configured to:
process at least one frame of the series of frames, captured by the imaging system, by at least a first modeling technique relative to a first image attribute and obtain a corresponding first product identification probability that an item, captured within the at least one frame of the series of frames, is estimated to be a first product;
process one or more frames of the series of frames by a second modeling technique relative to a second image attribute that is different than the first image attribute, and obtain a corresponding second product identification probability that the item, captured within the one or more frames of the series of frames, is estimated to be the first product;
collectively evaluate the first product identification probability and the second product identification probability; and
identify, based on the collective evaluation, there is a threshold consistency relative to at least the first product identification probability and the second product identification probability and confirm an identification of the item as the first product based on the identification of the threshold consistency.

### Claim 2 (depends on 1)

The system of claim 1, wherein the decision control circuit is further configured to:
process a first set of frames of the series of frames by the first modeling technique relative to the first image attribute obtaining a set of first product identification probabilities comprising the first product identification probability;
determine an aggregated first identification probability of the first product as a function of the set of the first product identification probabilities corresponding to the first set of frames, and wherein the confirming the threshold consistency comprises confirming the threshold consistency based on the aggregated first identification probability.

### Claim 3 (depends on 2)

The system of claim 2, wherein the decision control circuit is further configured to:
process a second set of frames of the series of frames by the second modeling technique relative to the second image attribute obtaining a set of second product identification probabilities comprising the second product identification probability;
determine an aggregated second identification probability of the first product as a function of the set of the second product identification probabilities corresponding to the second set of frames, and wherein the confirming the threshold consistency comprises identifying the threshold consistency based on the aggregated first identification probability and the aggregated second product identification probability.

### Claim 4 (depends on 1)

The system of claim 1, wherein the second modeling technique comprises a barcode recognition modeling technique and the second image attribute comprises a barcode image attribute, wherein the decision control circuit is configured to process each frame of the series of frames by the barcode recognition modeling technique relative to the barcode image attribute, and obtain corresponding barcode product identification probabilities that the item, captured within each frame of the series of frames, is estimated to be the first product.

### Claim 5 (depends on 1)

The system of claim 1, wherein the first modeling technique comprises optical character recognition (OCR) modeling technique and the decision control circuit in processing the at least one frame of the series of frames according to the first modeling technique is configured to process each frame of the series of frames by the OCR modeling technique relative to text image attributes, and obtain corresponding text product identification probabilities that the item, captured within each frame of the series of frames, is estimated to be the first product.

### Claim 6 (depends on 1)

The system of claim 1, further comprising:
apply a first weighting relative to first product identification probabilities as a function of an expected degree of accuracy relative to the first image attribute to provide at least one weighted first product identification probability;
apply a second weighting relative to second product identification probabilities as a function of an expected degree of accuracy relative to the second image attribute to provide at least one weighted second product identification probability;
identify when there is the threshold consistency between the at least one weighted first product identification probability and the at least one weighted second product identification probability.

### Claim 7 (depends on 6)

The system of claim 6, wherein the decision control circuit is configured to identify a threshold inconsistency between at least one of first product identification probabilities and one of second product identification probabilities, and apply at least one of a first weighting to at least one of the first product identification probabilities and a second weighting to at least one of the second product identification probabilities in response to identifying the threshold inconsistency.

### Claim 8 (depends on 6)

The system of claim 6, wherein the decision control circuit, in applying the second weighting to the second product identification probabilities, is configured to identify in each frame of the series of frames a number of textual words present on the item and multiply the number of textual words by a word multiplier in defining the second weighting.

### Claim 9 (depends on 1)

The system of claim 1, wherein the decision control circuit is further configured to:
process multiple first product identification probabilities and obtain an aggregated first identification probability;
process multiple second product identification probabilities and obtain an aggregated second identification probability;
apply a first weighting to the aggregated first identification probability as a function of an expected degree of accuracy relative to the first image attribute to provide a weighted first product identification probability, apply a second weighting to the aggregated second identification probability as a function of an expected degree of accuracy relative to the second image attribute to provide a weighted second product identification probability, and determine a resultant identification probability of the first product based on the weighted first product identification probability and the weighted second product identification probability.

### Claim 10 (depends on 1)

The system of claim 1, further comprising:
add the first product to a virtual cart; and
initiate a checkout and obtain payment for each product within the virtual cart.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
cause a generation of an order corresponding to the virtual cart, obtain a dynamically generated optical machine-readable representation of the order.

### Claim 12 (depends on 10)

The system of claim 10, further comprising:
obtain a confirmation of payment; and
display the confirmation of payment.

### Claim 13

A method to recognize products in a physical facility, comprising:
controlling an imaging system to capture a series of frames;
receiving the series of frames;
processing at least one frame of the series of frames by at least a first modeling technique relative to a first image attribute and obtaining a corresponding first product identification probability that an item, captured within the at least one frame of the series of frames, is estimated to be a first product;
processing one or more frames of the series of frames by a second modeling technique relative to a second image attribute that is different than the first image attribute and obtaining a second product identification probability that the item, captured within the one or more frames of the series of frames, is estimated to be the first product;
collectively evaluating the first product identification probability and second product identification probability; and
identifying, based on the collective evaluation, there is a threshold consistency relative to at least the first product identification probability and the second product identification probability and confirming an identification of the item as the first product based on the identification of the threshold consistency; and
causing an image of the first product to be displayed in response to identifying the threshold consistency.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
processing a first set of frames of the series of frames by the first modeling technique relative to the first image attribute obtaining a set of first product identification probabilities comprising the first product identification probability;
determining an aggregated first identification probability of the first product as a function of the set of the first product identification probabilities corresponding to the first set of frames; and
wherein the confirming the threshold consistency comprises confirming the threshold consistency based on the aggregated first identification probability.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
processing a second set of frames of the series of frames by the second modeling technique relative to the second image attribute obtaining a set of second product identification probabilities comprising the second product identification probability;
determining an aggregated second identification probability of the first product as a function of the set of the second product identification probabilities corresponding to the second set of frames; and
wherein the confirming the threshold consistency comprises identifying the threshold consistency based on the aggregated first identification probability and the aggregated second product identification probability.

### Claim 16 (depends on 13)

The method of claim 13, wherein the second modeling technique comprises a barcode recognition modeling technique and the second image attribute comprises a barcode image attribute;
processing each frame of the series of frames by the barcode recognition modeling technique relative to the barcode image attribute; and
obtaining corresponding barcode product identification probabilities that the item, captured within each frame of the series of frames, is estimated to be the first product.

### Claim 17 (depends on 13)

The method of claim 13, wherein the processing at least one from frame of the series of frames by the first modeling technique comprises processing each frame of the series of frames by an optical character recognition (OCR) modeling technique relative to text image attributes; and
obtaining corresponding text product identification probabilities that the item, captured within each frame of the series of frames, is estimated to be the first product.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
applying a first weighting relative to first identification probabilities as a function of an expected degree of accuracy relative to the first image attribute to provide at least one weighted first product identification probability;
applying a second weighting relative to second identification probabilities as a function of an expected degree of accuracy relative to the second image attribute to provide at least one weighted second product identification probability;
identifying when there is the threshold consistency between the at least one weighted first product identification probability and the at least one weighted second product identification probability.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
identifying a threshold inconsistency between at least one of first product identification probabilities and one of second product identification probabilities; and
applying at least one of a first weighting to at least one of the first product identification probabilities and a second weighting to at least one of the second product identification probabilities in response to identifying the threshold inconsistency.

### Claim 20 (depends on 18)

The method of claim 18, wherein the applying the second weighting to the second product identification probabilities comprises identifying in each frame of the series of frames a number of textual words present on the item; and
multiplying the number of textual words by a word multiplier in defining the second weighting.