# US 2021/0240996 A1 · Systems and methods for compliance monitoring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240996 A1
- Document ID: US20210240996A1
- Status: Published application
- Kind: A1
- Application number: 17/064,361
- Filed: 2020-10-06
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Element Ai Inc.
- Applicants: Element Ai Inc.
- Inventors: Negin SOKHANDAN ASL
- Web page: https://patentgrep.com/patent/US20210240996A1

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/2431: Multiple classes
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 30/0185: {Product, service or business identity fraud}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for monitoring product placement. The method comprises accessing a first image depicting a plurality of items arranged in accordance with a first layout, accessing a second image, the second image depicting at least some of the plurality of items arranged in accordance with a second layout. The method then proceeds to inputting, to a machine learning algorithm (MLA), a first density map and a second density map, the first density map having been generated from the first image and the second density map having been generated from the second image. An anomaly map is then outputted by the ML, the anomaly map comprising a first indication of an item class associated with an anomaly and a second indication of a position associated with the anomaly.

## Claims

### Claim 1

A computer-implemented method of multiclass features compliance monitoring, the method comprising:
accessing a first image, the first image depicting a plurality of features arranged in accordance with a first layout, the plurality of features being categorised in accordance with a plurality of feature classes;
accessing a second image, the second image depicting at least some of the plurality of features arranged in accordance with a second layout;
generating, a first density map from the first image, the first density map comprising a first plurality of density map layers, each one of the first plurality of density map layers being associated with a first set of distinct feature classes;
generating, a second density map from the second image, the second density map comprising a second plurality of density map layers, each one of the second plurality of density map layers being associated with a second set of distinct feature classes, the first and second set of distinct feature classes having at least some feature classes in common;
inputting, to an MLA, the first density map and the second density map, the MLA having been trained for generating one or more anomaly maps from density maps; and
outputting, by the MLA, an anomaly map generated by interleaving at least some of the density map layers of the first density map with density map layers of the second density map based on feature classes in common, the anomaly map comprising a first indication of a feature class associated with an anomaly and a second indication of a position associated with the anomaly.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
inputting, to a classifier, the anomaly map; and
outputting, by the classifier, a third indication of an anomaly type associated with the anomaly.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
outputting, the item class, the position and the anomaly type.

### Claim 4 (depends on 1)

The method of claim 1, wherein the MLA has been trained based on a loss function that establishes a norm of a difference between generated density maps and generated anomaly maps with corresponding ground truth maps.

### Claim 5 (depends on 1)

The method of claim 1, wherein the plurality of features comprise a plurality of product items categorised in accordance with a plurality of product item classes.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first density map and the second density map comprise Gaussian representations reflective of a probability of existence of a feature, each one of the Gaussian representations comprising a center and a standard deviation, the center being representative of a center of the feature and the standard deviation being representative of a size of the feature.

### Claim 7 (depends on 1)

The method of claim 1, wherein the MLA comprises a convolutional neural network (CNN), the CNN comprising a first group of layers configured so as to increase a number of channels and decrease a spatial size of the first and second density maps and a second group of layers configured so as to decrease a number of channels and increase a spatial size of the first and second density maps.

### Claim 8

A computer-implemented method of monitoring product placement compliance, the method comprising:
accessing a first image, the first image depicting a plurality of items arranged in accordance with a first layout, the plurality of items being categorised in accordance with a plurality of item classes, the first layout defining a reference state of placement of the items;
accessing a second image, the second image depicting at least some of the plurality of items arranged in accordance with a second layout, the second layout defining a different state of placement of the items;
inputting, to a machine learning algorithm (MLA), a first density map and a second density map, the first density map having been generated from the first image and the second density map having been generated from the second image, the MLA having been trained for generating one or more anomaly maps from density maps; and
outputting, by the MLA, an anomaly map, the anomaly map comprising a first indication of an item class associated with an anomaly and a second indication of a position associated with the anomaly.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
inputting, to a classifier, the anomaly map; and
outputting, by the classifier, a third indication of an anomaly type associated with the anomaly.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
outputting the item class, the position and the anomaly type of the anomaly.

### Claim 11 (depends on 8)

The method of claim 8, wherein the MLA is a first MLA and wherein the first density map has been generated by inputting the first image to a second MLA, the first density map comprising a first plurality of density map layers, each one of the first plurality of density map layers being associated with a distinct item class.

### Claim 12 (depends on 11)

The method of claim 11, wherein the second density map has been generated by inputting the second image to the second MLA, the second density map comprising a second plurality of density map layers, each one of the second plurality of density map layers being associated with a distinct item class.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first plurality of density map layers and the second plurality of density map layers are interleaved before being inputted to the first MLA.

### Claim 14 (depends on 12)

The method of claim 12, wherein the anomaly map comprises a third plurality of density map layers, each one of the third plurality of density map layers being associated with a distinct item class.

### Claim 15 (depends on 8)

The method of claim 8, wherein the first density map and the second density map comprise Gaussian representations reflective of a probability of existence of an item, each one of the Gaussian representations comprising a center and a standard deviation, the center being representative of a center of the item and the standard deviation being representative of a size of the item.

### Claim 16 (depends on 8)

The method of claim 8, wherein the MLA comprises a convolutional neural network (CNN), the CNN comprising a first group of layers configured so as to increase a number of channels and decrease a spatial size of the first and second density maps and a second group of layers configured so as to decrease a number of channels and increase a spatial size of the first and second density maps.

### Claim 17

A system for multiclass features compliance monitoring, the system comprising:
at least one processor, and
memory storing a plurality of executable instructions which, when executed by the at least one processor, cause the system to:
access a first image, the first image depicting a plurality of features arranged in accordance with a first layout, the plurality of features being categorised in accordance with a plurality of feature classes;
access a second image, the second image depicting at least some of the plurality of features arranged in accordance with a second layout;
generate, a first density map from the first image, the first density map comprising a first plurality of density map layers, each one of the first plurality of density map layers being associated with a first set of distinct feature classes;
generate, a second density map from the second image, the second density map comprising a second plurality of density map layers, each one of the second plurality of density map layers being associated with a second set of distinct feature classes, the first and second set of distinct feature classes having at least some feature classes in common;
input, to an MLA, the first density map and the second density map, the MLA having been trained for generating one or more anomaly maps from density maps; and
output, by the MLA, an anomaly map generated by interleaving at least some of the density map layers of the first density map with density map layers of the second density map based on feature classes in common, the anomaly map comprising a first indication of a feature class associated with an anomaly and a second indication of a position associated with the anomaly.

### Claim 18 (depends on 17)

The system of claim 17, wherein the plurality of features comprise a plurality of product items categorised in accordance with a plurality of product item classes.

### Claim 19 (depends on 17)

The system of claim 17, wherein the first density map and the second density map comprise Gaussian representations reflective of a probability of existence of a feature, each one of the Gaussian representations comprising a center and a standard deviation, the center being representative of a center of the feature and the standard deviation being representative of a size of the feature.

### Claim 20 (depends on 17)

The system of claim 17, wherein the MLA comprises a convolutional neural network (CNN), the CNN comprising a first group of layers configured so as to increase a number of channels and decrease a spatial size of the first and second density maps and a second group of layers configured so as to decrease a number of channels and increase a spatial size of the first and second density maps.

### Claim 21

A system for monitoring product placement compliance, the system comprising:
at least one processor, and
memory storing a plurality of executable instructions which, when executed by the at least one processor, cause the system to:
access a first image, the first image depicting a plurality of items arranged in accordance with a first layout, the plurality of items being categorised in accordance with a plurality of item classes, the first layout defining a reference state of placement of the items;
access a second image, the second image depicting at least some of the plurality of items arranged in accordance with a second layout, the second layout defining a different state of placement of the items;
input, to a machine learning algorithm (MLA), a first density map and a second density map, the first density map having been generated from the first image and the second density map having been generated from the second image, the MLA having been trained for generating one or more anomaly maps from density maps; and
output, by the MLA, an anomaly map, the anomaly map comprising a first indication of an item class associated with an anomaly and a second indication of a position associated with the anomaly.
