# US 12,223,729 B2 · System and method for surveillance of goods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,729 B2
- Document ID: US12223729B2
- Status: Granted
- Kind: B2
- Application number: 18/161,694
- Filed: 2023-01-30
- Published: 2023-08-03
- Granted: 2025-02-11
- Earliest claimed: 2022-02-02
- Assignees: Everseen Limited
- Applicants: Everseen Limited
- Inventors: Calin Barburescu; Robert Preda; Alexandra Laicu; Ana Cristina Todoran; Paul Iusztin; Marius Badea; Laura Vasilie; Alexandru Hulea; Qintilian-Emanuel Vintila
- Web page: https://patentgrep.com/patent/US12223729B2

## CPC

- G06Q 10/08: Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06K 7/1417: {2D bar codes}
- G06Q 10/083: Shipping
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08741: {by picking of items from inventory for fulfillment}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/44: {Event detection}
- G06T 2207/30196: Human being; Person
- G06T 2207/30242: Counting objects in image
- G06V 2201/07: Target detection
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An order checking system includes video sensors configured to capture video footage of a monitored area located in proximity to a receipt/dispatch portal. A processing unit performs event analysis on the captured video footage, detects an entity and from a door opening event, an incoming delivery from a third-party supplier, identifies the third-party supplier and implements a check-in process for delivery persons associated therewith, detects an ingress/egress of merchandise through the receipt/dispatch portal and validates that detected delivered products matches with data regarding products that the third-party supplier should be delivering. A database stores at least a dataset of face images/logos for detecting faces/brands and a dataset of product images for identification of products. The database records an outcome of an order checking process and a check out of a delivery person at an end of a delivery.

## Claims

### Claim 1

An order checking system comprising:
a plurality of video sensors configured to capture video footage of a monitored area located within an order receiving area or an order dispatch area of a receipt/dispatch portal;
a processing unit configured to:
perform event analysis on the captured video footage;
detect an entity in the video footage captured by the video sensors;
detect from a door opening event in the captured video footage, an incoming delivery from a third-party supplier;
identify the third-party supplier and implement a check-in process for delivery persons from the third-party supplier;
detect an ingress/egress of merchandise through the receipt/dispatch portal and validate that detected delivered products matches with data regarding products that the third-party supplier should be delivering;
analyse the video footage captured by the video sensors to determine if the receipt/dispatch portal is in an open state or a closed state using a Door State Detector;
analyse the video footage captured by the video sensors to detect if a delivery person has arrived at the receipt/dispatch portal using a Person Detector;
track movements of the delivery person from the captured video footage upon detection of the delivery person by the Person Detector using a Person Tracker;
detect a presence of a quick response (QR) code in the captured video footage and to read the QR code using a quick response (QR) Detector,
wherein the QR detector compares the detected QR code with known pre-approved QR codes for third-party suppliers/delivery persons to find a match, and
wherein if a match is found, the QR detector is configured to:
classify the delivery person presenting the QR code as an authorized entrant to an order fulfillment facility/order receiving facility; and
grant the delivery person access to the order fulfillment facility/order receiving facility; and
check one or more contents of merchandise to be delivered from, or received into, a premises of the order fulfillment facility/order receiving facility using a Pallet Monitor Module, wherein the Pallet Monitor Module comprises:
a Pallet Detector Module configured to detect a pallet;
a Pallet Tracker Module configured to track the detected pallet;
a Merchandise Classification per Pallet Module configured to classify merchandise on the detected pallet;
a Pallet Volume Estimator configured to estimate a quantity of merchandise on the detected pallet; and
an IN-OUT counter configured to extract information about a total number of pallets passing through the receipt/dispatch portal; and
a database communicably coupled to the processing unit, the database configured to store:
at least a dataset of face images/logos for use in detecting faces/brands;
a dataset of product images for use in identification of products; and
record an outcome of an order checking process and a check out of a delivery person at an end of a delivery for future retrieval upon request to the processing unit.

### Claim 2 (depends on 1)

The order checking system of claim 1, wherein the Person Detector and the Door State Detector is embodied as a neural network of a pre-determined architecture that is configured for person and door detection.

### Claim 3 (depends on 1)

The order checking system of claim 1, wherein the processing unit further comprises an Event Management module in communication with the Delivery Detection Module and the Pallet Monitor Module, the Event Management module comprising:
an Alert Manager configured to issue alerts concerning the detection of an authorized entrant and information about the merchandise that is being supplied/delivered during a supply/delivery episode; and
an Event Recorder configured to record the supply/delivery episode.

### Claim 4 (depends on 3)

The order checking system of claim 3, wherein the Alert Manager is further configured to issue alerts when:
the receipt/dispatch portal is left open for a period exceeding a first pre-defined threshold;
the pallet remains in a certain area for a period exceeding a second pre-defined threshold;
the contents of the pallet from a certain third-party supplier do not match corresponding particulars of an advance shipment notice;
the pallet exits the receipt/dispatch portal without being pre-registered;
a height of the pallet exceeds a pre-defined maximum allowed height;
an employee of the order fulfillment facility/order receiving facility is not present on arrival of the delivery person; and
the delivery person enters the order fulfillment facility/order receiving facility without previously signing in.

### Claim 5

A method for performing video surveillance, the method comprising:
capturing video footage, using a plurality of video sensors, of a monitored area located within an order receiving area or an order dispatch area of a receipt/dispatch portal;
performing, using a processing unit, event analysis on the captured video footage;
detecting, using the processing unit, an entity in the video footage captured by the video sensors;
detecting from a door opening event in the captured video footage, an incoming delivery from a third-party supplier using the processing unit;
identifying, using the processing unit, the third-party supplier and implementing a check-in process for delivery persons from the third-party supplier;
detecting, using the processing unit, an ingress/egress of merchandise through the receipt/dispatch portal and validating that detected delivered products matches with data regarding products that the third-party supplier should be delivering;
analysing the video footage captured by the video sensors to determine whether the receipt/dispatch portal is in an open state or a closed state;
analysing the video footage captured by the video sensors of the order checking system to detect whether a delivery person has arrived at the receipt/dispatch portal;
tracking movements of the delivery person from the captured video footage upon their detection;
detecting a presence of a quick response (QR) code in the captured video footage; and
reading the detected QR code as output;
comparing the detected QR code with known pre-approved QR codes for third-party suppliers/delivery persons to find a match, and wherein in the event a match is found:
classifying a delivery person presenting the QR code as an authorized entrant to an order fulfillment facility/order receiving facility; and
granting the delivery person access to the order fulfillment facility/order receiving facility;
checking one or more contents of merchandise to be delivered from, or received into, a premises of the order fulfillment facility/order receiving facility, wherein the step of checking comprises:
detecting a pallet using a Pallet Detector Module;
tracking the detected pallet using a Pallet Tracker Module;
classifying merchandise on the detected pallet using a Merchandise Classification per Pallet Module;
estimating a quantity of merchandise on the detected pallet using a Pallet Volume Estimator; and
extracting information about a total number of pallets passing through the receipt/dispatch portal using an IN-OUT counter;
storing, using a database, at least a dataset of face images/logos for use in detecting faces/brands and a dataset of product images for use in identification of products;
recording an outcome, using the database of an order checking process and a check out of a delivery person at an end of a delivery; and
retrieving the recording, by the processing unit from the database, upon request to the processing unit.

### Claim 6 (depends on 5)

The method of claim 5, further comprising implementing a neural network of a pre-determined architecture configured for person and door detection.

### Claim 7 (depends on 5)

The method of claim 5, further comprising:
issuing an alert concerning a detection of an authorized entrant and information about the merchandise that is being supplied/delivered during a supply/delivery episode using an Alert Manager; and
recording the supply/delivery episode using an Event Recorder.

### Claim 8 (depends on 7)

The method of claim 7, further comprising issuing alerts by the alert manager in the event:
the receipt/dispatch portal is left open for a period exceeding a first pre-defined threshold;
the pallet remains in a certain area for a period exceeding a second pre-defined threshold;
the contents of the pallet from a certain third-party supplier do not match the corresponding particulars of an advance shipment notice;
the pallet exits the receipt/dispatch portal without being pre-registered;
a height of the pallet exceeds a pre-defined maximum allowed height;
an employee of the order fulfillment facility/order receiving facility is not present on arrival of the delivery person; and
the delivery person enters the order fulfillment facility/order receiving facility without previously signing in.

### Claim 9

A non-transitory computer readable medium having stored thereon computer-executable instructions which, when executed by a processing unit, causes the processing unit to:
capture video footage, using a plurality of video sensors, of a monitored area located within an order receiving area or an order dispatch area of a receipt/dispatch portal;
perform event analysis on the captured video footage;
detect an entity in the video footage captured by the video sensors;
detect, from a door opening event in the captured video footage, an incoming delivery from a third-party supplier using the processing unit;
identify the third-party supplier and implementing a check-in process for delivery persons from the third-party supplier;
detect an ingress/egress of merchandise through the receipt/dispatch portal and validating that detected delivered products matches with data regarding products that the third-party supplier should be delivering;
analyse the video footage captured by the video sensors of the order checking system to determine whether the receipt/dispatch portal is in an open state or a closed state;
analyse the video footage captured by the video sensors to detect whether a delivery person has arrived at the receipt/dispatch portal;
track movements of the delivery person from the captured video footage upon their detection;
detect a presence of a quick response (QR) code in the captured video footage and read the detected QR code as output;
compare the detected QR code with known pre-approved QR codes for third-party suppliers/delivery persons to find a match, and wherein in the event a match is found:
classify a delivery person presenting the QR code as an authorized entrant to an order fulfillment facility/order receiving facility; and
grant the delivery person access to the order fulfillment facility/order receiving facility;
check one or more contents of merchandise to be delivered from, or received into, a premises of the order fulfillment facility/order receiving facility, wherein the step of checking comprises:
detecting a pallet using a Pallet Detector Module;
tracking the detected pallet using a Pallet Tracker Module;
classifying merchandise on the detected pallet using a Merchandise Classification per Pallet Module;
estimating a quantity of merchandise on the detected pallet using a Pallet Volume Estimator; and
extracting information about a total number of pallets passing through the receipt/dispatch portal using an IN-OUT counter;
store, using a database, at least a dataset of face images/logos for use in detecting faces/brands and a dataset of product images for use in identification of products; and
record, using the database, an outcome of an order checking process and a check out of a delivery person at an end of a delivery; and
retrieve the recording from the database upon request by the processing unit.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein upon execution of the executable instructions, the processing unit is configured to:
issue an alert concerning a detection of an authorized entrant and information about the merchandise that is being supplied/delivered during a supply/delivery episode using an Alert Manager; and
record the supply/delivery episode using an Event Recorder.

### Claim 11 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein upon execution of the executable instructions, the processing unit is configured to issue alerts using the alert manager in the event:
the receipt/dispatch portal is left open for a period exceeding a first pre-defined threshold;
the pallet remains in a certain area for a period exceeding a second pre-defined threshold;
the contents of a pallet from a certain third-party supplier do not match corresponding particulars of an advance shipment notice;
the pallet exits the receipt/dispatch portal without being pre-registered;
a height of the pallet exceeds a pre-defined maximum allowed height;
an employee of the order fulfillment facility/order receiving facility is not present on arrival of the delivery person arrives; and
the delivery person enters the order fulfillment facility/order receiving facility without previously signing in.