# US 2017/0286773 A1 · Planogram Assisted Inventory System and Method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286773 A1
- Document ID: US20170286773A1
- Status: Published application
- Kind: A1
- Application number: 15/471,813
- Filed: 2017-03-28
- Published: 2017-10-05
- Earliest claimed: 2016-03-29
- Assignees: Bossa Nova Robotics IP, Inc.
- Applicants: Bossa Nova Robotics IP, Inc.
- Inventors: Sarjoun Skaff; Jonathan Davis Taylor; Stephen Vincent Williams; Simant Dube
- Web page: https://patentgrep.com/patent/US20170286773A1

## CPC

- G01S 17/88: Lidar systems specially adapted for specific applications
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G05D 1/0246: {using a video camera in combination with image processing means}
- G06F 18/24: Classification techniques
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/64: Three-dimensional [3D] objects
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A manually assisted robot inventory monitoring method provides for detecting and reading shelf labels using an autonomous robot. Bounding boxes around possible products in a panoramic image can be taken with at least one camera associated with the autonomous robot. Products in the bounding boxes are automatically identified, with those that are not being later manually identified.

## Claims

### Claim 1

A planogram assisted inventory method, comprising the steps of:
providing an initial planogram with identified products;
creating a panoramic image including products on a shelf, with possible products being surrounded by bounding boxes;
detecting and reading shelf labels to localize and associate possible products with both the initial planogram and the possible products in the bounding boxes; and
identifying the possible products with a product classifier.

### Claim 2 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
segmenting images to extract product images and identifiers and associate with the initial planogram.

### Claim 3 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
template matching possible products in the bounding boxes without product segmentation.

### Claim 4 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
using vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.

### Claim 5 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
using deep learning method to build classifiers that identify products or gaps.

### Claim 6 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
allowing an autonomous robot to move beside the shelf, with the autonomous robot acting as a movable base capable of both autonomously detecting and reading shelf labels and creating the panoramic image.

### Claim 7 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of
allowing an autonomous robot to move beside a shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and of products positioned on shelves using 3D cameras, a LIDAR system or structure from motion and multiple cameras.

### Claim 8 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of using manual input to identify possible products designated in the bounding boxes.

### Claim 9 (depends on 1)

The planogram assisted inventory method of claim 1, wherein the product classifier can include at least one of a machine learning system or a deep learning system.

### Claim 10 (depends on 1)

The planogram assisted inventory method of claim 1, further comprising the step of identifying stock depletion level for a product using a product depth detection system.

### Claim 11

A manually assisted robot inventory monitoring method, comprising the steps of:
detecting and reading shelf labels using an autonomous robot;
defining bounding boxes around possible products in a panoramic image taken with at least one camera associated with the autonomous robot;
automatically identifying products in the bounding boxes; and
manually identifying products in the bounding boxes that were not automatically identified.

### Claim 12 (depends on 11)

The manually assisted robot inventory monitoring method of claim 11, further comprising the step of using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

### Claim 13 (depends on 11)

The manually assisted robot inventory monitoring method of claim 11, wherein the bounding boxes can surround gaps between possible products.

### Claim 14 (depends on 11)

The planogram assisted inventory method of claim 11, wherein the step of automatically identifying products can further comprise use of at least one of a machine learning or a deep learning system.

### Claim 15 (depends on 11)

The planogram assisted inventory method of claim 11, further comprising the step of identifying stock depletion level for an automatically or manually identified product using a product depth detection system.

### Claim 16

An inventory system comprising:
a planogram with identified products;
an autonomous robot able to capture a panoramic image including products on a shelf, with possible products being identified by bounding boxes;
a shelf label detector and reader attached to the autonomous robot and able to localize and associate possible products with both the planogram and the possible products in the bounding boxes; and
a product classifier to use information from the planogram and the possible products in the bounding boxes to identify products in the panoramic image.

### Claim 17 (depends on 16)

The inventory system of claim 16, further comprising
an image segmentation system able to extract product images and identifiers and associate them with the planogram.

### Claim 18 (depends on 16)

The inventory system of claim 16, further comprising
a template matching system to match possible products in the bounding boxes without product segmentation.

### Claim 19 (depends on 16)

The inventory system of claim 16, further comprising
a system for vision based product recognition to extract features descriptors for comparison with possible products in the bounding boxes.

### Claim 20 (depends on 16)

The inventory system of claim 16, further comprising
a deep learning system to build classifiers that identify products.
