# US 2018/0322448 A1 · System and method for automatically restocking items on shelves

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322448 A1
- Document ID: US20180322448A1
- Status: Published application
- Kind: A1
- Application number: 15/971,308
- Filed: 2018-05-04
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Walmart Apollo, LLC
- Applicants: Walmart Apollo, LLC
- Inventors: Behzad Nemati; Ehsan Nazarian
- Web page: https://patentgrep.com/patent/US20180322448A1

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08726: {for replenishment processing, procedures, or recommendations using forecasting or optimisation}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06Q 20/203: {Inventory monitoring}
- G06Q 20/208: {Input by product or record sensing, e.g. weighing or scanner processing}
- G06Q 30/00: Commerce
- G06T 7/0008: {checking presence/absence}
- G06V 20/20: in augmented reality scenes
- G06V 20/47: {Detecting features for summarising video content}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G07G 1/0036: {Checkout procedures}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G07G: REGISTERING THE RECEIPT OF CASH, VALUABLES, OR TOKENS

## Abstract

Systems, methods and computer-readable media for automating the restocking of shelves process by sending a notification when a product on a shelf has reached, or will reach, an undesired level of emptiness. This is determined using imaging sensors, such as cameras, which can calculate how full or empty a respective shelf is and predict when the shelf will need to be restocked. When the restocking time arrives, the notification can be sent to automated systems, which automatically cause new products to be stocked on the shelf, or can be sent to human beings (such as store associates) who can then perform the restocking.

## Claims

### Claim 1

A method comprising:
receiving, from a plurality of imaging sensors, real-time imaging data, the real-time imaging data including images of a plurality of products on a store shelf, the plurality of products having a plurality of product types, wherein the plurality of imaging sensors:
(1) detect motion between an imaging sensor in the plurality of imaging sensors and the store shelf;
(2) detect an end of the motion; and
(3) wait a pre-determined amount of time after the end of the motion before recording the images;
storing the real-time imaging data in a database, wherein the database contains item-specific data associated with each product in the plurality of products, the item-specific data including dimensions, weight, and orientation information of the each product;
calculating, in real-time and for each product in the plurality of products, a current quantity of a product on the store shelf based on the real-time imaging data and the item-specific data;
calculating a current depletion rate for each product in the plurality of products based on the current quantity of the each product, a previous quantity of the each product, and a historical sales rate of the each product;
forecasting when an emptiness threshold for each product in the plurality of products will be reached based on the current depletion rate for the each product and the current quantity of the each product, to yield a plurality of forecasted replenishment times, each forecasted replenishment time in the plurality of forecasted replenishment times identifying when a respective product in the plurality of products will reach the emptiness threshold; and
at each forecasted replenishment time in the plurality of forecasted replenishment times:
generating an alert for restocking the respective product; and
triggering restocking of the respective product based on the alert.

### Claim 2 (depends on 1)

The method of claim 1, wherein the predetermined time is in a range of between about 10-20 seconds.

### Claim 3 (depends on 1)

The method of claim 1, wherein the forecasting further comprises using machine learning to forecast the plurality of forecasted replenishment times.

### Claim 4 (depends on 3)

The method of claim 3, wherein the machine learning is updated on a periodic basis based on actual sales of the plurality of products.

### Claim 5 (depends on 1)

The method of claim 1, wherein each imaging sensor in the plurality of imaging sensors comprises a camera that is positioned across an aisle from the store shelf.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of imaging sensors further capture a duration of the motion, and wherein capturing of the images only occurs when the duration of the motion exceeds a motion duration threshold.

### Claim 7 (depends on 1)

The method of claim 1, wherein the plurality of imaging sensors are dormant until an aisle entrance motion detector detects a presence of an individual on an aisle containing the store shelf.

### Claim 8

A restocking system, comprising:
a plurality of imaging sensors configured to capture imaging data of items on store shelves;
a database having item-specific data stored, the item-specific data including dimensions, weight, and orientation information of the items;
a processor; and
a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
storing the imaging data in the database;
calculating a current quantity of a product on the store shelves based on the imaging data and the item-specific data;
calculating a current depletion rate for the product based on the current quantity of the product, a previous quantity of the product, and a historical sales rate of the product;
forecasting when an emptiness threshold for the product will be reached based on the current depletion rate and the current quantity of the product, to yield a forecasted replenishment time;
at the forecasted replenishment time, generating an alert for restocking the product; and
triggering restocking of the product based on the alert.

### Claim 9 (depends on 8)

The restocking system of claim 8, wherein the database stores a predefined image of the store shelves in a full state.

### Claim 10 (depends on 8)

The restocking system of claim 8, wherein the plurality of imaging sensors:
(1) detect motion between an imaging sensor in the plurality of imaging sensors and the store shelf;
(2) detect an end of the motion; and
(3) wait a pre-determined amount of time after the end of the motion before recording the images.

### Claim 11 (depends on 10)

The restocking system of claim 10, wherein the predetermined time is in a range of between about 10-20 seconds.

### Claim 12 (depends on 9)

The restocking system of claim 9, wherein the forecasting further comprises using machine learning to forecast the plurality of forecasted replenishment times.

### Claim 13 (depends on 12)

The restocking system of claim 12, wherein the machine learning is updated on a periodic basis based on actual sales of the plurality of products.

### Claim 14 (depends on 9)

The restocking system of claim 9, wherein each imaging sensor in the plurality of imaging sensors comprises a camera that is positioned across an aisle from the store shelves.

### Claim 15 (depends on 9)

The restocking system of claim 9, wherein the plurality of imaging sensors are dormant until an aisle entrance motion detector detects a presence of an individual on an aisle containing the store shelves.

### Claim 16

A non-transitory computer-readable storage medium having instructions stored which, when executed by a processor, cause the processor to perform operations comprising:
receiving, from a plurality of imaging sensors, real-time imaging data, the real-time imaging data including images of a plurality of products on a store shelf, the plurality of products having a plurality of product types, wherein the plurality of imaging sensors:
(1) detect motion between an imaging sensor in the plurality of imaging sensors and the store shelf;
(2) detect an end of the motion; and
(3) wait a pre-determined amount of time after the end of the motion before recording the images;
storing the real-time imaging data in a database, wherein the database contains item-specific data associated with each product in the plurality of products, the item-specific data including dimensions, weight, and orientation information of the each product;
calculating, in real-time and for each product in the plurality of products, a current quantity of a product on the store shelf based on the real-time imaging data and the item-specific data;
calculating a current depletion rate for each product in the plurality of products based on the current quantity of the each product, a previous quantity of the each product, and a historical sales rate of the each product;
forecasting when an emptiness threshold for each product in the plurality of products will be reached based on the current depletion rate for the each product and the current quantity of the each product, to yield a plurality of forecasted replenishment times, each forecasted replenishment time in the plurality of forecasted replenishment times identifying when a respective product in the plurality of products will reach the emptiness threshold; and
at each forecasted replenishment time in the plurality of forecasted replenishment times:
generating an alert for restocking the respective product; and
triggering restocking of the respective product based on the alert.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the predetermined time is in a range of between about 10-20 seconds.

### Claim 18 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein the forecasting further comprises using machine learning to forecast the plurality of forecasted replenishment times.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the machine learning is updated on a periodic basis based on actual sales of the plurality of products.

### Claim 20 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein each imaging sensor in the plurality of imaging sensors comprises a camera that is positioned across an aisle from the store shelf.
