# US 2020/0005225 A1 · On-shelf image based out-of-stock detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0005225 A1
- Document ID: US20200005225A1
- Status: Published application
- Kind: A1
- Application number: 16/024,519
- Filed: 2018-06-29
- Published: 2020-01-02
- Earliest claimed: 2018-06-29
- Assignees: Focal Systems, Inc.
- Applicants: Focal Systems, Inc.
- Inventors: Francois Chaubard
- Web page: https://patentgrep.com/patent/US20200005225A1

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- H04N 23/57: Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
- G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting}
- G06T 11/23: using straight lines or curves
- G06V 30/1473: {Recognising objects as potential recognition candidates based on visual cues, e.g. shapes}
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30232: Surveillance
- G06T 2210/12: Bounding box
- G06V 30/10: Character recognition
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An out-of-stock detection system notifies store management that a product is out of stock. The system captures images of a shelf and determines the position product labels thereon. For each product label, a bounding box is generated based on the position of each product label on the shelf. The system then identifies a product for each product label based on information within each product label and, for each product label, stores a product identified for each bounding box. Accordingly, the system performs an out-of-stock detection process that includes capturing additional image data of the shelf periodically that includes each bounding box, providing a portion of the additional image data for each bounding box to a model trained to determine whether the bounding box contains products, sending a notification for a product determined to be out of stock to a store client device based on output from the model.

## Claims

### Claim 1

A method comprising:
capturing first image data of a product display area in a store, the product display area including a plurality of product labels, each product label including product information describing a product associated with the product label;
determining, from the image data, a position of each of the plurality product labels within the product display area;
for each product label, generating a bounding box within the product display area based on the position of each product label within the product display area;
identifying a product associated with each product label based on the product information associated with each product label;
for each product label, storing an association of the identified product with the bounding box for the product label; and
performing an out-of-stock detection process that comprises:
capturing second image data of the product display area that includes each bounding box,
providing area portion of the second image data corresponding to each bounding box to a classifier model that is trained to determine whether the bounding box contains products,
determining whether one or more of the products is in the bounding box is out-of-stock based on the output of the classifier model for the bounding box associated with the product, and
sending a notification for one or more of the products determined to be out-of-stock to a store client device.

### Claim 2 (depends on 1)

The method of claim 1, wherein associated with each product label includes:
extracting the product information from each product label by applying an optical character recognition algorithm to the first image data; and
matching the product information to a database of products carried by the store.

### Claim 3 (depends on 1)

The method of claim 1, wherein identifying the product associated with the bounding box as out-of-stock includes identifying one or more empty portions within the bounding box.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first image data and the second image data are captured by a camera mounted on the product display area.

### Claim 5 (depends on 4)

The method of claim 4, wherein the first image data is captured at a higher resolution relative to the second image data.

### Claim 6 (depends on 4)

The method of claim 4, wherein the camera includes a motor allowing the camera to rotate 360 degrees to capture images of display areas on each of two opposing sides of the display area on which the camera is mounted.

### Claim 7 (depends on 1)

The method of claim 1, wherein the product display area corresponds to a shelf unit and an aisle comprises a plurality of shelf units, and wherein the camera is mounted on a movable track on top of the plurality of shelf units comprising an aisle in the store, the camera capturing an image of each product display area along the aisle and sending the captured image with a position data for the camera along the movable track to identify the product display area.

### Claim 8

A non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
receive first image data of a product display area in a store from a camera, the product display area including a plurality of product labels;
determine, from the image data, a position of each of the plurality product labels within the product display area;
generate, for each product label, a bounding box within the product display area based on the position of each product label within the product display area;
identify a product associated with each product label;
associate, for each product label, the identified product with the bounding box for the product label; and
perform an out-of-stock detection process that comprises:
receiving second image data of the product display area from the camera, the second image data being captured a different time relative to the first image data corresponding to image data for each bounding box identified using the first image data;
providing the extracted area corresponding to each bounding box to a classifier trained to determine presence of products within each bounding box; and
responsive to the classifier identifying one or more voids within a bounding box, identifying a product associated with the bounding box as out-of-stock; and
sending a restocking notification for the identified product to a store client device.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein identifying the product associated with each product label comprises:
obtaining information from a planogram of the store, the planogram identifying products associated with one or more product display areas within the store; and
matching, for at least a subset of bounding boxes of the product display area, the product in a particular bounding box the extracted product information from each product label to identify the product associated with the product label based on a location within the store associated with the product display area.

### Claim 10 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein identifying the product associated with each product label comprises:
extracting the product information from each product label by applying an optical character recognition algorithm to the first image data; and
matching the product information to a database of products carried by the store.

### Claim 11 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the first image data and the second image data are captured by a camera mounted on the product display area.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the first image data is captured at a higher resolution relative to the second image data.

### Claim 12 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein identifying the product associated with the bounding box as out-of-stock includes identifying one or more empty portions within the bounding box.

### Claim 12 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the camera includes a motor allowing the camera to rotate 360 degrees to capture images of display areas on each of two opposing sides of the display area on which the camera is mounted.

### Claim 13 (depends on 8)

The non-transitory computer-readable medium of claim 8, wherein the product display area corresponds to a shelf unit and an aisle comprises a plurality of shelf units, and wherein the camera is mounted on a movable track on top of the plurality of shelf units comprising an aisle in the store, the camera capturing an image of each product display area along the aisle and sending the captured image with a position data for the camera along the movable track to identify the product display area.

### Claim 14

A non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
obtain, at a first time, first image data of a product display area in a store at a first camera resolution, the product display area including a plurality of product labels;
generate a bounding box for each of the plurality of product labels of the product display area based on a position of each product label within the product display area;
identify a product associated with each product label;
store, for each product label, information the identified product with the bounding box for the product label;
receive, after the first time, second image data of the product display area at a second camera resolution, the second image data corresponding to image data for each bounding box identified using the first image data, the first camera resolution being higher relative to the second camera resolution;
provide area of second image data corresponding to each bounding box to a classifier trained to determine presence of products within each bounding box; and
responsive to the classifier the bounding box is less than full, identify a product associated with the bounding box as being out-of-stock; and
send a restocking notification for the identified product to a store client device.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein identifying the product associated with each product label comprises:
obtaining information from a planogram of the store, the planogram identifying products associated with one or more product display areas within the store; and
matching, for at least a subset of bounding boxes of the product display area, the product in a particular bounding box the extracted product information from each product label to identify the product associated with the product label based on a location within the store associated with the product display area.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein identifying the product associated with each product label comprises:
extracting the product information from each product label by applying an optical character recognition algorithm to the first image data; and
matching the product information to a database of products carried by the store.

### Claim 17 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the first image data and the second image data are captured by a camera mounted on the product display area.

### Claim 18 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein identifying the product associated with the bounding box as out-of-stock includes identifying one or more empty portions within the bounding box.

### Claim 19 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein the camera includes a motor allowing the camera to rotate 360 degrees to capture images of display areas on each of two opposing sides of the display area on which the camera is mounted.

### Claim 20 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein the product display area corresponds to a shelf unit and an aisle comprises a plurality of shelf units, and wherein the camera is mounted on a movable track on top of the plurality of shelf units comprising an aisle in the store, the camera capturing an image of each product display area along the aisle and sending the captured image with a position data for the camera along the movable track to identify the product display area.
