# US 2021/0241214 A1 · Method for tracking and maintaining promotional states of slots in inventory structures within a store

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241214 A1
- Document ID: US20210241214A1
- Status: Published application
- Kind: A1
- Application number: 17/169,326
- Filed: 2021-02-05
- Published: 2021-08-05
- Earliest claimed: 2020-02-05
- Assignees: Simbe Robotics, Inc.
- Applicants: Simbe Robotics, Inc.
- Inventors: Bradley Bogolea; Jariullah Safi; Lorin Vandegrift; Shiva Reddy; Tom Gehani
- Web page: https://patentgrep.com/patent/US20210241214A1

## CPC

- G06Q 30/0248: {Avoiding fraud}
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/08726: {for replenishment processing, procedures, or recommendations using forecasting or optimisation}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06Q 20/201: {Price look-up processing, e.g. updating}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G05D 2105/93: {for inventory}
- G05D 2107/67: {Shopping areas}
- G05D 2109/10: Land vehicles
- G06Q 10/08744: {by distributed inventory management}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

One variation of a method for tracking promotional states of slots in inventory structures within a store includes: accessing an image of an inventory structure within a store; detecting a shelf tag on the inventory structure in the image; extracting a set of features from the shelf tag detected in the image; detecting a promotional tag on the inventory structure in the image; extracting a set of features from the promotional tag detected in the image; detecting a deviation between the shelf tag and the promotional tag based on a difference between the sets of features; and, in response to detecting the deviation between the shelf tag and the promotional tag, identifying the first promotional tag as erroneous, and notifying a store associate to replace the first promotional tag with a second promotional tag at the first slot, the second promotional tag correcting the difference.

## Claims

### Claim 1

A method for tracking promotional states of slots in inventory structures within a store comprising:
accessing an image of an inventory structure within the store;
detecting a first shelf tag on the inventory structure in the image;
extracting a first set of features from the first shelf tag detected in the image;
identifying a first slot, in the inventory structure, corresponding to the first shelf tag;
detecting a first promotional tag on the inventory structure in the image;
extracting a second set of features from the first promotional tag detected in the image;
associating the first promotional tag with the first shelf tag;
detecting a deviation between the first shelf tag and the first promotional tag based on a difference between the first set of features and the second set of features;
in response to detecting the deviation between the first shelf tag and the first promotional tag:
identifying the first promotional tag as erroneous; and
generating a prompt to replace the first promotional tag with a second promotional tag at the first slot, the second promotional tag correcting the difference; and
transmitting the prompt to a computing device associated with a store associate.

### Claim 2 (depends on 1)

The method of claim 1:
further comprising:
retrieving a promotional tag rule for the store, the promotional tag rule specifying an arrangement of promotional tags relative to corresponding shelf tags and threshold distances between promotional tags and corresponding shelf tags; and
detecting a shelf face in the image;
wherein detecting the first shelf tag on the inventory structure comprises detecting a row of shelf tags on the shelf face;
wherein detecting the first promotional tag on the inventory structure comprises detecting a row of promotional tags on the shelf face; and
wherein associating the first promotional tag with the first shelf tag comprises:
characterizing a first distance between and a first relative position of the first promotional tag, in the row of promotional tags, and the first shelf tag, in the row of shelf tags; and
linking the first promotional tag to the first shelf tag in response to the first distance and the first relative position fulfilling the promotional tag rule.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
identifying a first product type specified by the first shelf tag based on the first set of features;
extracting a first promotion status advertised by the first promotional tag based on the second set of features; and
temporarily updating pricing data for the first product type in a point-of-sale system associated with the store according to the first promotion status advertised by the first promotional tag.

### Claim 4 (depends on 3)

The method of claim 3:
wherein transmitting the prompt to the computing device comprises transmitting the prompt to the computing device at a first time;
further comprising:
dispatching a robotic system to capture a second image of the inventory structure at a second time succeeding the first time;
detecting the first shelf tag on the inventory structure in the second image;
detecting a second promotional tag on the inventory structure in the second image;
extracting a third set of features from the second promotional tag detected in the image;
associating the second promotional tag with the first shelf tag;
detecting correspondence between the first shelf tag and the second promotional tag based on alignment of the first set of features and the third set of features; and
in response to detecting correspondence between the first shelf tag and the second promotional tag:
verifying replacement of the first promotional tag by the second promotional tag; and
updating pricing data for the first product type in the point-of-sale system according to a second promotion status advertised by the second promotional tag.

### Claim 5 (depends on 1)

The method of claim 1:
wherein extracting the second set of features from the first promotional tag comprises extracting a distribution of feature types from a first region of the image depicting the first promotional tag;
further comprising:
accessing a set of promotional tag templates;
matching the first promotional tag to a first promotional tag template, in the set of promotional tag templates, based on the distribution of feature types;
retrieving a target promotion status location and a promotion status interpreter for the first promotional tag template;
identifying a target feature, in the second set of features, located in the target promotion status location on the first promotional tag; and
interpreting a first promotion status advertised by the first promotional tag based on the target feature and the promotion status interpreter; and
wherein detecting the deviation between the first shelf tag and the first promotional tag comprises detecting the deviation in response to the first promotion status advertised by the first promotional tag exceeding a price advertised by the first shelf tag.

### Claim 6 (depends on 1)

The method of claim 1:
further comprising:
based on the first set of features, identifying a first product identifier of a first product type advertised by the first shelf tag;
retrieving a first attribute of the first product type from a product database based on the first product identifier; and
based on the second set of features, identifying a second attribute advertised by the first promotional tag; and
wherein detecting the deviation between the first shelf tag and the first promotional tag comprises detecting the deviation between the first shelf tag and the first promotional tag based on a conflict between the first attribute and the second attribute.

### Claim 7 (depends on 6)

The method of claim 6:
wherein retrieving the first attribute of the first product type from the product database comprises, based on the first product identifier, retrieving the first attribute comprising a first nutritional value of the first product type from the product database;
wherein identifying the second attribute advertised by the first promotional tag comprises, based on the second set of features, identifying the second attribute comprising a second nutritional value advertised by the first promotional tag; and
wherein detecting the deviation between the first shelf tag and the first promotional tag comprises detecting the deviation between the first shelf tag and the first promotional based on a conflict between the first nutritional value and the second nutritional value.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
identifying a product type specified by the first shelf tag based on the first set of features;
interpreting a target quantity of units of the product type assigned to the first slot based on the first set of features;
extracting a third set of features from a region of the image depicting the first slot;
identifying a quantity of product units of the product type occupying the first slot based on the third set of features;
in response to the quantity of product units differing from the target quantity by more than a threshold difference, generating a second prompt to restock the first slot with product units of the product type; and
transmitting the second prompt to the computing device associated with the store associate.

### Claim 9

A method for tracking promotional states of slots in inventory structures within a store comprising:
accessing an image of an inventory structure within a store;
detecting a first slot in the inventory structure depicted in the image;
identifying a first product type assigned to the first slot;
retrieving, from a promotion schedule, a current promotion status assigned to the first product type;
detecting a first promotional tag in the image;
associating the first promotional tag with the first slot;
extracting a first set of features from the first promotional tag detected in the image;
interpreting a first promotion status advertised by the first promotional tag based on the first set of features;
detecting a conflict between the first promotion status and the current promotion status;
in response to detecting the conflict:
flagging the first promotional tag; and
generating a notification to investigate the first promotional tag at the first slot; and
transmitting the notification to a computing device associated with a store associate.

### Claim 10 (depends on 9)

The method of claim 9:
wherein extracting the first set of features from the first promotional tag comprises extracting a distribution of feature types from a first region of the image depicting the first promotional tag; and
wherein interpreting the first promotion status advertised by the first promotional tag comprises:
accessing a set of promotional tag templates;
matching the first promotional tag to a first promotional tag template, in the set of promotional tag templates, based on the distribution of feature types;
retrieving a target promotion status location and a promotion status interpreter for the first promotional tag template;
identifying a target feature, in the set of features, located in the target promotion status location on the first promotional tag; and
interpreting the first promotion status advertised by the first promotional tag based on the target feature and the promotion status interpreter.

### Claim 11 (depends on 9)

The method of claim 9:
wherein retrieving the current promotion status assigned to the first product type comprises retrieving a target promotion format assigned to the first product type;
wherein interpreting the first promotion status advertised by the first promotional tag comprises interpreting a first promotion format advertised by the first promotional tag; and
wherein detecting the conflict between the first promotion status and the current promotion status comprises detecting a difference between the first promotion format and the target promotion format.

### Claim 12 (depends on 11)

The method of claim 11:
wherein retrieving the target promotion format assigned to the first product type comprises retrieving the target promotion format comprising “buy [X] quantity, get [Y] quantity free” from the promotion schedule; and
wherein interpreting the first promotion format advertised by the first promotional tag comprises interpreting the first promotion format comprising “[X]% discount” from the first set of features.

### Claim 13 (depends on 9)

The method of claim 9, further comprising, in response to detecting the conflict:
generating a print file for a replacement promotional tag based on the first promotion status and correcting the conflict; and
queuing a printer to print the replacement promotional tag.

### Claim 14 (depends on 9)

The method of claim 9:
wherein accessing the image of the inventory structure comprises accessing the image of the inventory structure captured on a current date;
wherein retrieving the current promotion status assigned to the first product type comprises retrieving the first promotion status specifying a date range for a promotion assigned to the first product type;
wherein detecting the conflict between the first promotion status and the current promotion status comprises detecting the conflict in response to the current data falling outside of the date range for the promotion; and
wherein generating the notification to investigate the first promotional tag at the first slot comprises generating the notification to remove the first promotional tag from the first slot.

### Claim 15 (depends on 9)

The method of claim 9:
wherein detecting the first slot in the inventory structure comprises locating the first slot in the inventory structure based on a position of the first shelf tag detected in the image; and
wherein identifying the first product type assigned to the first slot comprises:
detecting a first shelf tag on the inventory structure in the image;
extracting a second set of features from the first shelf tag detected in the image;
identifying a first product identifier of the first product type advertised by the first shelf tag based on the second set of features; and
identifying the first product type based on the product identifier.

### Claim 16 (depends on 15)

The method of claim 15:
further comprising:
interpreting a first price for the product type advertised by the first promotional tag based on the first set of features;
identifying a second price for the product type advertised by the first shelf tag based on the second set of features; and
verifying price consistency of the first promotional tag in response to the second price advertised by the first shelf tag exceeding the first price advertised by the first promotional tag; and
wherein detecting the conflict between the first promotion status and the current promotion status comprises, in response to verifying price consistency of the first promotional tag, scanning the first promotion status advertised by the first promotional tag for deviation from the current promotion status assigned to the first product type.

### Claim 17 (depends on 9)

The method of claim 9, further comprising:
scanning a digital product catalog defining a population of product types stocked in the store for current promotion statuses of the population of product types; and
compiling current promotion statuses for the population of product types, detected in the digital product catalog, into the promotion schedule.

### Claim 18 (depends on 9)

The method of claim 9:
wherein transmitting the notification to the computing device comprises transmitting the notification to the computing device at a first time;
further comprising:
temporarily updating pricing data for the first product type in a point-of-sale system associated with the store according to the first promotion status advertised by the first promotional tag.
dispatching a robotic system to capture a second image of the inventory structure at a second time succeeding the first time;
detecting a second promotional tag on the inventory structure in the second image;
associating the second promotional tag with the first slot;
extracting a third set of features from the second promotional tag detected in the image;
interpreting a second promotion status advertised by the second promotional tag based on the third set of features;
detecting correspondence between the second promotion status and the current promotion status; and
in response to detecting correspondence between the second promotion status and the current promotion status:
verifying replacement of the first promotional tag by the second promotional tag; and
updating pricing data for the first product type in the point-of-sale system according to the second promotion status advertised by the second promotional tag.

### Claim 19

A method for tracking promotional states of slots in inventory structures within a store comprising:
accessing an image of an inventory structure within a store;
detecting a set of shelf faces in the inventory structure in the image;
detecting a shelf tag on a first shelf face, in the set of shelf faces, in the image;
delineating a slot above the first shelf face based on a position of the shelf tag in the image;
identifying a product type assigned to the slot;
retrieving, from a promotion schedule, a current promotion status associated with the product type;
detecting an absence of a promotional tag indicating the current promotion status at the slot in the image;
in response to detecting the absence of the promotional tag, generating a notification to install the promotional tag at the slot; and
transmitting the notification to a computing device associated with a store associate.

### Claim 20 (depends on 19)

The method of claim 19:
wherein identifying the product type assigned to the slot comprises:
extracting a first set of features from the shelf tag detected in the image; and
identifying the product type assigned to the slot based on the first set of features; and
further comprising:
interpreting a target quantity of units of the product type assigned to the slot based on the first set of features;
extracting a second set of features from a region of the image depicting the slot;
identifying a quantity of product units of the product type occupying the slot based on the second set of features;
in response to the quantity of product units differing from the target quantity by more than a threshold difference, generating a second notification to restock the slot with product units of the product type; and
transmitting the second notification to the computing device associated with the store associate.