# US 2017/0286901 A1 · System and Method for Locating, Identifying and Counting Items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286901 A1
- Document ID: US20170286901A1
- Status: Published application
- Kind: A1
- Application number: 15/471,722
- Filed: 2017-03-28
- Published: 2017-10-05
- Earliest claimed: 2016-03-29
- Assignees: Bossa Nova Robotics IP, Inc.
- Applicants: Bossa Nova Robotics IP, Inc.
- Inventors: Sarjoun Skaff; Jonathan Davis Taylor; Stephen Vincent Williams; Simant Dube
- Web page: https://patentgrep.com/patent/US20170286901A1

## CPC

- G01S 17/88: Lidar systems specially adapted for specific applications
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G05D 1/0246: {using a video camera in combination with image processing means}
- G06F 18/24: Classification techniques
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06V 10/245: {by locating a pattern; Special marks for positioning}
- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/64: Three-dimensional [3D] objects
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system for building a product library without requiring a planogram includes an image capture unit operated to provide images of items. Taking data from a shelf label detector and depth map creation unit, a processing module can be used to compare detected shelf labels to a depth map, define a product bounding box, and associate the bounding box with an image provided by the image capture unit to build image descriptors. The system can include one or more autonomous robots for supporting and moving the image capture unit and other components.

## Claims

### Claim 1

A inventory monitoring method, comprising the steps of:
detecting and reading shelf labels on at least one shelf;
defining one or more bounding boxes around potential inventory in an image taken with at least one camera; and
associating the bounding boxes in the image to shelf labels to build at least one of a product classifier or a product library.

### Claim 2 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base capable of autonomously reading shelf labels and defining multiple bounding boxes.

### Claim 3 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base having multiple cameras and a LIDAR range sensing system.

### Claim 4 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and products positioned on shelves.

### Claim 5 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of building a product library with a product scope containing at least one or more of the following: one product identifier; one or more sets of descriptors; confidence levels for each set of descriptors; dates at which each set of descriptors was generated; one or more shelf position metric estimates; confidence levels for each shelf position metric estimate; one or more shelf position topological estimates; a count number for each shelf position topological estimate; one or more image templates of the product; and dimensions of the product.

### Claim 6 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

### Claim 7 (depends on 1)

The inventory monitoring method of claim 1, wherein bounding boxes around both potential inventory and gaps between potential inventory are automatically defined.

### Claim 8 (depends on 1)

The inventory monitoring method of claim 1, wherein bounding boxes are manually associated with at least one of a shelf label or an on-product marker.

### Claim 9 (depends on 1)

The inventory monitoring method of claim 1, wherein bounding boxes are automatically associated with at least one of a shelf label or an on-product marker.

### Claim 10 (depends on 1)

The inventory monitoring method of claim 1, further comprising the step of building the product library without use of an initial planogram.

### Claim 11

A system for building a product classifier, comprising:
an image capture unit;
a shelf label detector and reader to capture shelf label data;
a processing module to define multiple bounding boxes, and
a trainable image classifier to associate images within the bounding boxes with the shelf label data, the images being provided by the image capture unit.

### Claim 12 (depends on 11)

The system of claim 11, wherein the image capture unit is mounted on an autonomous robot and is connected to the shelf label detector and reader.

### Claim 13 (depends on 11)

The system of claim 11, wherein the trainable image classifier is trained using machine learning.

### Claim 14 (depends on 11)

The system of claim 11, wherein the trainable image classifier is trained using a deep learning system.

### Claim 15

A inventory monitoring method, comprising the steps of:
detecting and reading shelf labels;
detecting distance from at least one camera to at least one shelf;
defining bounding boxes around potential inventory in an image taken with at least one camera; and
associating selected bounding boxes in the image to selected shelf labels to build at least one of a product classifier or a product library.

### Claim 16 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base capable of detecting distance, reading shelf labels, and defining bounding boxes.

### Claim 17 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base having multiple cameras and at least one of a LIDAR range sensing system or a depth camera.

### Claim 18 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of allowing an autonomous robot to move beside the at least one shelf, with the autonomous robot acting as a movable base to capture a depth map of shelves and of products positioned on shelves.

### Claim 19 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of building a product library with a product scope containing at least one or more of the following: one product identifier; one or more sets of descriptors; confidence levels for each set of descriptors; dates at which each set of descriptors was generated; one or more shelf position metric estimates; confidence levels for each shelf position metric estimate; one or more shelf position topological estimates; a count number for each shelf position topological estimate; one or more image templates of the product; and dimensions of the product.

### Claim 20 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of building a product library using manual input to identify potential inventory designated in the multiple bounding boxes in the image taken with at least one camera

### Claim 21 (depends on 15)

The inventory monitoring method of claim 15, wherein the bounding boxes can surround gaps between potential inventory.

### Claim 22 (depends on 15)

The inventory monitoring method of claim 15, further comprising the step of building the product library without use of an initial planogram.