# US 2024/0005263 A1 · Machine vision system for advancement of trailer loading/unloading visibility

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005263 A1
- Document ID: US20240005263A1
- Status: Published application
- Kind: A1
- Application number: 17/853,748
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: United Parcel Service of America, Inc.
- Applicants: United Parcel Service of America, Inc.
- Inventors: Youngjun CHOI; Po-Nien LIN; Hyunki LEE
- Web page: https://patentgrep.com/patent/US20240005263A1

## CPC

- G06Q 10/08355: {Routing methods}
- G06V 20/176: {Urban or other man-made structures}
- G06Q 10/109: Time management, e.g. calendars, reminders, meetings or time accounting
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 20/36: {Indoor scenes}
- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- B65G 67/20: Loading covered vehicles
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)

## Abstract

Provided are embodiments for providing analytics indicative of object detection or fill-level detection at or near real-time based on video data captured during an unloading or loading process. A computerized system may detect and classify, using an object-detection machine learning (ML) model, an object based on the video data. A computerized system may further determine, using a fill-level ML model, a fill-level of the storage compartment based on a comparison of edges of the storage compartment to a total dimension corresponding to the edge. In this manner, the various implementations described herein provide a technique for computing systems employing image processing and machine learning techniques to a video data stream to generate analytics associated with the unloading or loading process at or near real-time.

## Claims

### Claim 1

At least one computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations comprising:
accessing a video indicative of the inside of a storage compartment from which items are being loaded or unloaded;
extracting video data for the video based at least on the items being loaded or unloaded from inside of the storage compartment to produce a plurality of video data features comprising a fill-level data feature and an object-detection data feature;
determining, using a fill-level machine learning (ML) model, a fill-level associated with the inside of the storage compartment based on at least the video and the fill-level data feature of the plurality of video data features;
classifying, using an object-detection ML model, an object detected inside of the storage compartment based at least on the video and the object-detection data feature of the plurality of video data features;
generating analytics indicative of the determined fill-level and the classified object; and
communicating the analytics to a display device to cause the display device to generate a graphical user interface (GUI).

### Claim 2 (depends on 1)

The computer-storage media of claim 1, wherein the video is received from a monocular camera.

### Claim 3 (depends on 1)

The computer-storage media of claim 1, wherein the GUI comprises a stream region that comprises a live stream of the video and a fill-level indication.

### Claim 4 (depends on 1)

The computer-storage media of claim 1, wherein the GUI comprises an analytics region that comprises an indication of at least one of: a door open time, a door close time, a door open period, a human enter time, a human exit time, a maximum number of workers, a working hours, or a number of pallets.

### Claim 5 (depends on 1)

The computer-storage media of claim 1, wherein the fill-level ML model is trained, using supervised machine learning, based on labeled data indicative of an edge, a length of the edge, an orientation of the edge, or an area coordinate associated with the storage compartment.

### Claim 6 (depends on 1)

The computer-storage media of claim 1, wherein the object-detection ML model is trained, using supervised machine learning, based on labeled data indicative of a human object, a pallet object, a load stand object, a parcel retainer object, a parcel on a conveyer belt object, a forever bag object, the conveyer belt object, or a small container object.

### Claim 7 (depends on 1)

The computerized system of claim 1, wherein the analytics comprise:
a fill-level indicative of the items stored in the storage compartment; and
an object indication indicative of whether the object corresponds to a human object, a pallet object, a load stand object, a parcel retainer object, a parcel, a forever bag object, a conveyer belt object, or a small container object.

### Claim 8 (depends on 1)

The computerized system of claim 1, wherein the object-detection features comprise at least one of:
an object feature, comprising at least one of: a human object, a pallet object, a load stand object, a parcel retainer object, a parcel, a forever bag object, a conveyer belt object, or a small container object; or
a status feature, comprising at least one of: door open status, door closed status, door partially open status, door open but trailer not ready status, or unloading or loading completed status.

### Claim 9 (depends on 1)

The computerized system of claim 1, wherein the fill-level features comprise at least one of: edge detection, edge length, edge orientation, or area coordinates associated with the storage compartment.

### Claim 10

A computerized system, comprising:
at least one computer processor; and
computer memory storing computer-useable instructions that, when used by at least one computer processor, cause the at least one computer processor to perform operations comprising:
accessing a video indicative of the inside of a storage compartment from which items are being loaded or unloaded;
extracting video data for the video based at least on the items being loaded or unloaded from inside of the storage compartment to produce a plurality of video data features comprising a fill-level data feature and an object-detection data feature;
determining a fill-level associated with the inside of the storage compartment based on at least the fill-level data feature of the plurality of video data features;
detecting and classifying an object inside of the storage compartment based at least on the object-detection data feature of the plurality of video data features; and
causing presentation of a graphical user interface (GUI) comprising:
a stream region comprising a stream of the video; and
an analytics region comprising analytics indicative of progress in the items being loaded or unloaded from the storage compartment based on the determined fill-level and the object.

### Claim 11 (depends on 10)

The computerized system of claim 10, wherein causing presentation comprises:
updating the stream region to continue playing the stream of the video; and
removing an indication associated with the object when the object disappears from the video stream, or adding another indication when another object is classified.

### Claim 12 (depends on 10)

The computerized system of claim 10, wherein the stream region comprises an indication associated with the object, wherein the indication comprises a visually distinct symbol that classifies the object based on the classification.

### Claim 13 (depends on 10)

The computerized system of claim 10, wherein the indication associated with the object is determined based on a Kalman filter that compares a current detected state of the object with a predicted state of the object, wherein the state of the object corresponds to whether the object appears in the stream of the video.

### Claim 14 (depends on 10)

The computerized system of claim 10, wherein the video is received from a monocular camera positioned within or on a vehicle that comprises the storage compartment.

### Claim 15 (depends on 10)

The computerized system of claim 10, wherein the analytics indicative of progress comprises an indication corresponding to at least one of: a door open status, a door closed status, a door partially open status, a door open but trailer not ready status, or an unloading or loading completed status.

### Claim 16 (depends on 11)

The computerized system of claim 11, wherein the object comprises at least one of: a human object, a pallet object, a load stand object, a parcel retainer object, a parcel on a conveyer belt object, a forever bag object, the conveyer belt object, or a small container object.

### Claim 17

A computer-implemented method, comprising:
accessing a fill-level machine learning (ML) model and an object-detection ML model;
training the fill-level ML model based on a first set of labeled data corresponding to a fill-level data feature, of a plurality of video data features, indicative of a fill-level of items being loaded or unloaded from a storage compartment;
training the object-detection ML model based on a second set of labeled data corresponding to an object-detection data feature, of the plurality of video data features, indicative of a classified object within the storage compartment; and
deploying the trained object-detection ML model and the trained fill-level ML model to a computing device.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, further comprising:
validating the fill-level ML model based on a first set of unlabeled data applied to a corresponding loss function; and
validating the object-detection ML model based on a second set of unlabeled data applied to a corresponding loss function.

### Claim 19 (depends on 17)

The computer-implemented method of claim 17, wherein the first set of labeled data comprises edges, from a video frame, that have been identified as an intersection between surfaces defining an edge of an inside of the storage compartment.

### Claim 20 (depends on 17)

The computer-implemented method of claim 17, wherein the second set of labeled data comprises objects, from a video frame, that have been identified as at least one of: a human object, a pallet object, a load stand object, a parcel retainer object, a parcel on a conveyer belt object, a forever bag object, a conveyer belt object, or a small container object.
