# US 2024/0202282 A1 · Systems and methods for building image databases to facilitate computer vision-based consumer product identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202282 A1
- Document ID: US20240202282A1
- Status: Published application
- Kind: A1
- Application number: 18/584,954
- Filed: 2024-02-22
- Published: 2024-06-20
- Earliest claimed: 2020-08-27
- Assignees: Walmart Apollo, LLC
- Applicants: Walmart Apollo, LLC
- Inventors: Soren A. Larson; Soheil Salehian-Dardashti; Elizabeth J. Barton; Victoria Moeller-Chan
- Web page: https://patentgrep.com/patent/US20240202282A1

## CPC

- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 16/53: Querying
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/08726: {for replenishment processing, procedures, or recommendations using forecasting or optimisation}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/68: Food, e.g. fruit or vegetables
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for facilitating computer-vision-based detection and identification of consumer products comprise a support surface for supporting thereon one or more consumer products to be detected and a plurality of digital cameras each arranged at a different height relative to the support surface and positioned such that a field of view of each of the digital cameras includes the consumer product(s) located on the support surface. Movement of the support surface and/or the digital cameras permits the digital cameras to capture images of the consumer product from different angles of view and up to a full 360 degree of rotation around the at least one consumer product. A computing device uses at least the image data from such images to train a computer vision consumer product identification model to generate a reference image data model to be used for recognition of the at least one consumer product.

## Claims

### Claim 1

A system for facilitating computer-vision-based identification of consumer products, the system comprising:
a support surface for supporting thereon at least one consumer product to be identified;
at least one digital camera positioned such that:
a field of view of the at least one digital camera includes the at least one consumer product located on the support surface; and
the at least one camera is permitted to capture images of the at least one consumer product;
an electronic database configured to store electronic data including:
image data associated with the images of the at least one consumer product captured the at least one digital camera; and
identifier data associated with the at least one consumer product; and
a computing device in communication with the at least one digital camera and the electronic database, the computing device including a control circuit configured to:
obtain the image data and the identifier data from the electronic database; and
using the image data and the identifier data obtained by the computing device from the electronic database, train a computer vision consumer product identification model to generate a reference image data model to be used for recognition of the at least one consumer product when the at least one consumer product is located in a product storage unit.

### Claim 2 (depends on 1)

The system of claim 1, wherein the at least one digital camera is configured to move relative to the at least one consumer product located on the support surface to capture the images of the at least one consumer product from at least one angle.

### Claim 3 (depends on 2)

The system of claim 2, wherein the at least one digital camera is configured to capture from 1 to 10 frames per second.

### Claim 4 (depends on 1)

The system of claim 1, wherein the support surface is monochrome, and further comprising a monochrome background positioned around at least a portion of the support surface in the field of view of the at least one digital camera, wherein the monochrome support surface and the monochrome background increase capacity of the images captured by the at least one digital camera for background subtraction.

### Claim 5 (depends on 1)

The system of claim 1, further comprising at least one scanner including at least a first sensor configured to:
scan an identifier of the at least one consumer product; and
generate the identifier data for storage in the electronic database based on a scan of the identifier of the at least one consumer product by the scanner.

### Claim 6 (depends on 5)

The system of claim 5, wherein:
the at least one scanner further includes at least a second sensor configured to detect a fill level in an interior of the at least one consumer product and to generate fill level data for storage in the electronic database in association with the at least one consumer product;
the electronic database further stores reference fill level data indicating a fill level in the interior of the at least one consumer product when in an unconsumed state and when in a at least partially consumed state; and
the control circuit is programmed to correlate the fill level in the interior of the at least one consumer product detected by the at least one scanner to the reference fill level data to determine a level of consumption of the at least one consumer product relative to the fill level in the interior of the at least one consumer product when in the unconsumed state.

### Claim 7 (depends on 5)

The system of claim 5, wherein:
the at least one scanner includes at least a third sensor configured to detect a number of individual items in a pack of the at least one consumer product;
the electronic database further stores reference pack number data indicating a number of the individual items in the pack when the pack is in a fully stocked state; and
the control circuit is programmed to correlate the number of individual items in the pack of the at least one consumer product detected by the third sensor to the reference pack number data to determine how many of the individual items in the pack have been consumed relative to the number of the individual items in the pack when the pack is in a fully stocked state.

### Claim 8 (depends on 1)

The system of claim 1, wherein the control circuit of the computing device is further configured to:
obtain a first image of a plurality of consumer products located in the product storage unit captured by at least one digital camera located proximate the product storage unit;
detect each one of the plurality of individual consumer products in the first image; and
train a computer vision consumer product localization model to generate a reference bounding box model for drawing virtual boundary lines around each one of the individual consumer products located in the product storage unit.

### Claim 9 (depends on 8)

The system of claim 8, wherein the electronic database further stores the reference image data model and the reference bounding box model for the at least one consumer product in association with the identifier data associated with the at least one consumer product.

### Claim 10 (depends on 9)

The system of claim 9, wherein the computing device is further configured to:
obtain a second image of a consumer product located with other consumer products in the consumer product storage unit, the second image being captured by the at least one digital camera located proximate the product storage unit;
detect the individual consumer products located in the obtained image; and
correlate image data from the obtained second image to the reference image data model and the reference bounding box model to recognize an identity of each one of the individual consumer products detected in the obtained second image.

### Claim 11

A method of facilitating computer-vision-based identification of consumer products, the method comprising:
providing a support surface for supporting thereon at least one consumer product to be identified;
providing at least one digital camera having a field of view that includes the at least one consumer product located;
capturing images of the at least one consumer product via the at least one digital camera;
providing an electronic database configured to store electronic data including:
image data associated with the images of the at least one consumer product captured by the at least one digital camera; and
identifier data associated with the at least one consumer product; and
providing a computing device in communication with the at least one digital camera and the electronic database, the computing device including a control circuit; and
by the control circuit:
obtaining the image data and the identifier data from the electronic database; and
using the image data and the consumer product identifier data obtained by the computing device from the electronic database, training a computer vision consumer product identification model to generate a reference image data model for recognition of the at least one consumer product when the at least one consumer product is located in a product storage unit.

### Claim 12 (depends on 11)

The method of claim 11, further comprising moving the at least one digital camera relative to the at least one consumer product to capture the images of the at least one consumer product from at least one angle.

### Claim 13 (depends on 12)

The method of claim 12, further comprising capturing from 1 to 10 frames per second by the at least one digital camera the at least one consumer product located in the product storage unit.

### Claim 14 (depends on 11)

The method of claim 11, wherein the support surface is monochrome, and further comprising providing a monochrome background positioned around at least a portion of the support surface in the field of view of the at least one digital camera, wherein the monochrome support surface and the monochrome background increase capacity of the images captured by the at least one digital camera for background subtraction.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
providing at least one scanner including at least a first sensor;
scanning an identifier of the at least one consumer product via the at least one scanner; and
generating the identifier data for storage in the electronic database based on the scanning of the identifier of the at least one consumer product by the scanner.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
providing the at least one scanner with at least a second sensor;
via the at least one scanner, detecting a fill level in an interior of the at least one consumer product, and generating fill level data for storage in the electronic database in association with the at least one consumer product;
storing in the electronic database further reference fill level data indicating a fill level in the interior of the at least one consumer product when in an unconsumed state and when in a at least partially consumed state; and
by the control circuit, correlating the fill level in the interior of the at least one consumer product detected by the at least one scanner to the reference fill level data to determine a level of consumption of the at least one consumer product relative to the fill level in the interior of the at least one consumer product when in the unconsumed state.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
providing the at least one scanner with at least a third sensor:
detecting, via the at least one scanner, a number of individual items in a pack of the at least one consumer product;
storing in the electronic database reference pack number data indicating a number of the individual items in the pack when the pack is in a fully stocked state; and
by the control circuit, correlating the number of individual items in the pack of the at least one consumer product detected by the third sensor to the reference pack number data to determine how many of the individual items in the pack have been consumed relative to the number of the individual items in the pack when the pack is in a fully stocked state.

### Claim 18 (depends on 11)

The method of claim 11, further comprising, by the computing device:
obtaining a first image of a plurality of consumer products located in the product storage unit captured by at least one digital camera located proximate the product storage unit;
detecting each one of the plurality of individual consumer products in the first image; and
training a computer vision consumer product localization model to generate a reference bounding box model for drawing virtual boundary lines around each one of the individual consumer products located in the product storage unit.

### Claim 19 (depends on 18)

The method of claim 18, further comprising storing in the electronic database the reference image data model and the reference bounding box model for the at least one consumer product in association with the identifier data associated with the at least one consumer product.

### Claim 20 (depends on 19)

The method of claim 19, further comprising, by the computing device:
obtaining a second image of a consumer product located with other consumer products in the consumer product storage unit, the second image being captured by the at least one digital camera located proximate the product storage unit;
detecting the individual consumer products located in the obtained image; and
correlating image data from the obtained second image to the reference image data model and the reference bounding box model to recognize an identity of each one of the individual consumer products detected in the obtained second image.
