# US 12,118,499 B2 · Apparatuses, computer-implemented methods, and computer program products for automatic item searching and verification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,499 B2
- Document ID: US12118499B2
- Status: Granted
- Kind: B2
- Application number: 17/242,447
- Filed: 2021-04-28
- Published: 2021-12-30
- Granted: 2024-10-15
- Earliest claimed: 2020-05-20
- Assignees: Hand Held Products, Inc.
- Applicants: Hand Held Products, Inc.
- Inventors: Manjul Bizoara
- Web page: https://patentgrep.com/patent/US12118499B2

## CPC

- G06Q 10/083: Shipping
- G06Q 10/08: Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q 10/0833: Tracking
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/141: Control of illumination
- G06V 20/20: in augmented reality scenes
- G06V 20/63: {Scene text, e.g. street names}
- G06V 30/142: using hand-held instruments; Constructional details of the instruments
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments of the present disclosure provide for automatic item searching and verification. In example contexts, embodiments provide such functionality to improve the accuracy and efficiency of item searching during item loading and/or unloading, such as when delivering packages. Some embodiments provide for capturing an image, or set of images, identifying relevant items represented in the captured images, generating augmented reality elements associated with such relevant items, and rendering the augmented reality elements in an augmented reality environment, which can be maintained as the user continues to perform user action. Some embodiments additionally or alternatively detect improper actions based on captured image(s), and generate and provide notifications to a user for viewing.

## Claims

### Claim 1

An apparatus for automatic item searching and verification, the apparatus comprising at least one processor and at least one non-transitory memory including computer-coded instructions thereon, the computer-coded instructions, with the at least one processor, configure the apparatus to:
retrieve a target item data set associated with item distribution route data;
receive a captured image comprising an item representation set and an item information representation set associated with the item representation set,
wherein each item representation of the item representation set is associated with an item position in an augmented reality environment;
identify, from an item information representation of the item information representation set, a scanned item data object corresponding to the item information representation;
determine that the scanned item data object comprises a relevant item data object based on at least the target item data set;
generate an augmented reality element corresponding to the relevant item data object;
cause rendering, to a user interface, of the captured image augmented based on at least the augmented reality element,
wherein the augmented reality element is rendered based on at least a first item position associated with an item representation corresponding to the augmented reality element;
capture, using an imaging apparatus, a second captured image set comprising a second visual indicia set;
decode, from the second visual indicia set, a second scanned item data set associated with the second visual indicia set, the second scanned item data set representing a loaded item data set;
identify, based on at least the second scanned item data set, an undetected relevant item data object;
generate an undetected relevant item notification based on at least the undetected relevant item data object; and
cause rendering, to the user interface, of the undetected relevant item notification.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein to receive the captured image, the apparatus is configured to:
capture at least the captured image using the imaging apparatus.

### Claim 3 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
identify, from a second item information representation of the item information representation set, a second scanned item data object corresponding to a second item representation associated with the second item information representation;
identify the second scanned item data object comprises a second relevant item data object based on at least the target item data set associated with the item distribution route data, wherein the second relevant item data object corresponds to the second item representation;
generate a second augmented reality element corresponding to the second relevant item data object; and
cause rendering, to the user interface, of the captured image augmented based on at least the second augmented reality element, wherein the second augmented reality element is rendered based on a second item position associated with the second item representation corresponding to the second augmented reality element.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein to identify the scanned item data object corresponding to the item information representation, the apparatus is configured to:
detect visual indicia in the captured image; and
decode the visual indicia to identify the scanned item data object.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein to identify the scanned item data object corresponding to the item information representation, the apparatus is configured to:
detect an item label in the captured image; and
extract, using an optical character recognition image processing algorithm, the scanned item data object.

### Claim 6 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
detect, from the second scanned item data set, an improper item data object based on at least the target item data set;
generate an improper scanned item notification based on at least the improper item data object; and
cause rendering, to the user interface, of the improper scanned item notification.

### Claim 7 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
identify a device location, wherein the apparatus is configured to retrieve the target item data set based on at least the device location.

### Claim 8 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
identify a device location, wherein the device location is associated with at least a portion of the item distribution route data;
retrieve a second target item data set based on the item distribution route data and the device location, the second target item data set representing a target unload data set;
capture, using the imaging apparatus, a second captured image comprising a second item representation set and a second item information representation set associated with the second item representation set,
wherein each item representation in the second item representation set is associated with a second item position in a second augmented reality environment;
identify, from a second item information representation of the item information representation set, a second scanned item data object corresponding to the second item information representation;
identify the second scanned item data object comprises a second relevant item data object based on at least the second target item data set;
generate a second augmented reality element corresponding to the second relevant item data object; and
cause rendering, to the user interface, the second captured image augmented based on at least the second augmented reality element, wherein the second augmented reality element is rendered based on the second item position associated with a second item representation corresponding to the second augmented reality element.

### Claim 9 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
detect, from the second scanned item data set, an improper item data object based on the target item data set;
generate an improper scanned item notification based on at least the improper item data object; and
cause rendering, to the user interface, of the improper scanned item notification.

### Claim 10 (depends on 1)

The apparatus according to claim 1, the apparatus further configured to:
identify, based on at least the second scanned item data set, an improper unloaded item data object;
generate an improper unloaded item notification based on at least the improper unloaded item data object; and
cause rendering, to the user interface, of the improper unloaded item notification.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein the target item data set comprises a first target item data set associated with a first device location, and the apparatus further configured to:
identify a second device location, wherein the second device location is associated with at least a second portion of the item distribution route data;
retrieve a second target item data set based on at least the item distribution route data and the second device location;
capture, using the imaging apparatus, a second captured image comprising a second item representation set and a second item information representation set associated with the second item representation set;
identify, from a second item information representation of the second item information representation set, a second scanned item data object corresponding to a second item representation associated with the second item information representation;
identify the second scanned item data object comprises a missed item data object associated with the first device location;
generate a missed item notification based on at least the missed item data object associated with the first device location; and
cause rendering, to the user interface, of the missed item notification.

### Claim 12 (depends on 1)

The apparatus according to claim 1, wherein the target item data set is retrieved from a remote server over a first communications network via at least one application programming interface.

### Claim 13 (depends on 1)

The apparatus according to claim 1, wherein to receive the captured image, the apparatus is configured to capture the captured image using a multi-imager imaging apparatus.

### Claim 14

A computer-implemented method for automatic item searching and verification, the computer-implemented method comprising:
retrieving a target item data set associated with item distribution route data;
receiving a captured image comprising an item representation set and an item information representation set associated with the item representation set,
wherein each item representation of the item representation set is associated with an item position in an augmented reality environment;
identifying, from an item information representation of the item information representation set, a scanned item data object corresponding to the item information representation;
determining that the scanned item data object comprises a relevant item data object based on at least the target item data set;
generating an augmented reality element corresponding to the relevant item data object;
causing rendering, to a user interface, of the captured image augmented based on at least the augmented reality element,
wherein the augmented reality element is rendered based on at least a first item position associated with an item representation corresponding to the augmented reality element;
capturing, using an imaging apparatus, a second captured image set comprising a second visual indicia set;
decoding, from the second visual indicia set, a second scanned item data set associated with the second visual indicia set, the second scanned item data set representing a loaded item data set;
identifying, based on at least the second scanned item data set, an undetected relevant item data object;
generating an undetected relevant item notification based on at least the undetected relevant item data object; and
causing rendering, to the user interface, of the undetected relevant item notification.

### Claim 15 (depends on 14)

The computer-implemented method according to claim 14, the computer-implemented method further comprising:
identifying, from a second item information representation of the item information representation set, a second scanned item data object corresponding to a second item representation associated with the second item information representation;
identifying the second scanned item data object comprises a second relevant item data object based on at least the target item data set associated with the item distribution route data, wherein the second relevant item data object corresponds to the second item representation;
generating a second augmented reality element corresponding to the second relevant item data object; and
causing rendering, to the user interface, of the captured image augmented based on at least the second augmented reality element, wherein the second augmented reality element is rendered based on a second item position associated with the second item representation corresponding to the second augmented reality element.

### Claim 16 (depends on 14)

The computer-implemented method according to claim 14, wherein identifying the scanned item data object corresponding to the item information representation comprises:
detecting an item label in the captured image; and
extracting, using an optical character recognition image processing algorithm, the scanned item data object.

### Claim 17 (depends on 14)

The computer-implemented method according to claim 14, the computer-implemented method further comprising:
identifying a device location, wherein retrieving the target item data set is based on at least the device location.

### Claim 18 (depends on 14)

The computer-implemented method according to claim 14, wherein receiving the captured image comprises capturing the captured image using a multi-imager imaging apparatus.

### Claim 19

A computer program product for automatic item searching and verification, the computer program product comprising a non-transitory computer-readable storage medium having computer program code stored thereon, the computer program code in execution with at least one processor configured for:
retrieving a target item data set associated with item distribution route data;
receiving a captured image comprising an item representation set and an item information representation set associated with the item representation set,
wherein each item representation of the item representation set is associated with an item position in an augmented reality environment;
identifying, from an item information representation of the item information representation set, a scanned item data object corresponding to the item information representation;
determining that the scanned item data object comprises a relevant item data object based on at least the target item data set;
generating an augmented reality element corresponding to the relevant item data object;
causing rendering, to a user interface, of the captured image augmented based on at least the augmented reality element,
wherein the augmented reality element is rendered based on at least a first item position associated with an item representation corresponding to the augmented reality element;
capturing, using an imaging apparatus, a second captured image set comprising a second visual indicia set;
decoding, from the second visual indicia set, a second scanned item data set associated with the second visual indicia set, the second scanned item data set representing a loaded item data set;
identifying, based on at least the second scanned item data set, an undetected relevant item data object;
generate an undetected relevant item notification based on at least the undetected relevant item data object; and
cause rendering, to the user interface, of the undetected relevant item notification.