# US 12,002,008 B2 · Methods and apparatus for machine learning system for edge computer vision and active reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,008 B2
- Document ID: US12002008B2
- Status: Granted
- Kind: B2
- Application number: 18/404,377
- Filed: 2024-01-04
- Published: 2024-04-25
- Granted: 2024-06-04
- Earliest claimed: 2022-09-29
- Assignees: NOMAD Go, Inc.
- Applicants: NOMAD Go, Inc.
- Inventors: Jon Vogel; David Greschler; Jonah Friedl
- Web page: https://patentgrep.com/patent/US12002008B2

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G01S 17/89: for mapping or imaging
- G06Q 10/08724: {by shelf level inventory management, e.g. planograms}
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06T 7/50: Depth or shape recovery
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 20/64: Three-dimensional [3D] objects
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method includes receiving image frames of an inventory. The method includes locating a control point used to determine a spatial search in the image frames and detecting, by a machine learning model, a plurality of storage units in the spatial search. Each storage unit is associated with a unit type from a plurality of unit types. The method includes calculating a storage unit count from a plurality of storage unit counts and for each unit type from the plurality of unit types from the plurality of storage units detected based on depth analysis. Each storage unit count includes a total number of storage units associated with each unit type. The method includes determining a restock status of each unit type based on the storage unit count for each unit type, and automatically generating a replenish request based on the restock status.

## Claims

### Claim 1

An apparatus comprising:
a processor of a user device; and
a memory operatively coupled to the processor, the memory storing instructions to cause the processor to:
receive, from a sensor operatively coupled to the processor, a plurality of image frames of an inventory;
locate a control point used to determine a spatial search in the plurality of image frames;
detect, by a machine learning model, a plurality of storage units in the spatial search, each storage unit from the plurality of storage units associated with a unit type from a plurality of unit types;
calculate a storage unit count from a plurality of storage unit counts and for each unit type from the plurality of unit types from the plurality of storage units detected, based on depth calculation;
identify a duplicate storage unit based on the plurality of storage unit counts;
exclude the duplicate storage unit from the storage unit count associated with the duplicate storage unit;
determine a restock status of each unit type from the plurality of unit types based on the storage unit count for each unit type; and
automatically generate a replenish request based on the restock status.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the memory stores instructions to cause the processor to:
generate, via augmented reality, a plurality of icons for each storage unit from the plurality of storage units to be displayed on the user device, the plurality of icons including:
a storage unit label for each unit type from the plurality of unit types;
the storage unit count for each unit type from the plurality of unit types; and
a storage type color based on a storage type from a plurality of storage types for each unit type from the plurality of unit types; and
display the plurality of icons on the user device, the plurality of icons displayed in front of the plurality of storage units in the plurality of image frames.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the memory stores instructions to cause the processor to generate the plurality of icons for the control point and the spatial search.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein the memory stores instructions to cause the processor to adjust a placement of the plurality of icons in substantially real-time based on an orientation of the sensor.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein the memory storing instructions to cause the processor to generate the plurality of icons for the control point further includes instructions to cause the processor to generate a plurality of icons for a spatial label for each subset of storage units associated with each unit type.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the memory storing instructions to cause the processor to calculate the storage unit count for each unit type from the plurality of unit types further includes instructions to cause the processor to:
identify the spatial label for each subset of storage units from a plurality of subsets of storage units from the plurality of storage units; and
calculate, via the machine learning model, the storage unit count for each subset of storage units associated with each unit type.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the memory storing instructions to cause the processor to detect the plurality of storage units in the spatial search further includes instructions to cause the processor to:
identify, via the sensor, a storage unit identifier from a plurality of storage unit identifiers located on each storage unit from the plurality of storage units; and
calculate, via the machine learning model, the storage unit count for each unit type based on the storage unit identifier from the plurality of storage unit identifiers.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein:
the plurality of storage units is a first plurality of storage units;
the memory storing instructions to cause the processor to:
record the plurality of storage unit identifiers associated with the first plurality of storage units and the control point in the memory; and
identify a second plurality of storage units based on the plurality of storage unit identifiers stored in the memory based on the control point.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the memory storing instructions to cause the processor to calculate the storage unit count for each storage unit from the plurality of storage units further includes instructions to cause the processor to automatically calculate the storage unit count for each storage unit from the plurality of storage units.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the memory storing instructions to cause the processor to determine the restock status of each unit type from the plurality of unit types based on the storage unit count further includes instructions to cause the processor to automatically determine the restock status of each unit type from the plurality of unit types.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the sensor is not fixed and is configured to capture the plurality of image frames in substantially real time.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the control point is a quick-response (QR) code.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the memory stores instructions to cause the processor to:
generate a digital inventory dashboard based on the plurality of storage unit counts and the restock status of each unit type from the plurality of unit types; and
display the digital inventory dashboard on the user device.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the sensor is a LIDAR sensor.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the memory storing instructions to cause the processor to detect, by the machine learning model, the plurality of storage units in the spatial search further includes instructions to cause the processor to identify each storage unit, each storage unit not having a predefined orientation.

### Claim 16 (depends on 1)

The apparatus of claim 1, wherein the memory stores instructions to cause the processor to train the machine learning model using a training set to produce a trained machine learning model, the training set including an augmented storage unit digital model correlated to a storage unit identity.

### Claim 17 (depends on 1)

The apparatus of claim 1, wherein:
the unit type is a first unit type; and
the memory stores instructions to cause the processor to filter out a subset of storage units associated with a second unit type, prior to calculating the storage unit count for the first unit type.

### Claim 18

A method, comprising:
receiving, from a sensor operatively coupled to a processor of a user device, a plurality of image frames of an inventory;
locating a control point used to determine a spatial search in the plurality of image frames;
detecting, by a machine learning model, a plurality of storage units in the spatial search, each storage unit from the plurality of storage units associated with a unit type from a plurality of unit types;
calculating a storage unit count from a plurality of storage unit counts and for each unit type from the plurality of unit types from the plurality of storage units detected, based on depth calculation;
generating a digital model from a plurality of digital models, each digital model from the plurality of digital models being overlayed around a different storage unit from the plurality of storage units;
determining, based on at least an overlap between an area surrounding one or more digital models, a duplicate storage unit; and
updating the storage unit count associated with the duplicate storage unit.

### Claim 19 (depends on 18)

The method of claim 18, wherein the calculating the storage unit count includes:
identifying a plurality of columns of storage units of a same unit type from the plurality of unit types;
calculating, via depth calculation and for each column from the plurality of columns, a number of storage units in that column;
generating, for each column from the plurality of columns, an icon from a plurality of icons and that represents a count of storage units for that column; and
calculating the storage unit count for each unit type from the plurality of unit types based on a sum of the count of storage units for each column from the plurality of columns for that unit type.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
focusing, via the sensor, on the icon that represents the count of storage units for a column; and
in response to focusing on the icon for a predetermined time period, enabling a user to manually update the count of storage units for the column.

### Claim 21 (depends on 18)

The method of claim 18, further comprising:
generating, for each subset of digital models from the plurality of digital models and that is associated with a unit type from the plurality of unit types, a digital label that is overlayed on that subset of digital models, the digital label including a description of that subset of digital models;
focusing, via the sensor, on the digital label of a subset of digital models from the plurality of digital models; and
in response to focusing on the digital label for a predetermined time period, enabling a user to manually update the digital label.

### Claim 22 (depends on 18)

The method of claim 18, further comprising:
determining (1) a restock status of each unit type from the plurality of unit types based on the storage unit count for that unit type and (2) a planogram compliance, via the depth calculation, for each unit type from the plurality of unit types; and
automatically generating a replenish request based on the restock status and the planogram compliance for each unit type from the plurality of unit types.

### Claim 23 (depends on 18)

The method of claim 18, wherein the depth calculation is based on a stored depth value associated with each unit type from the plurality of unit types and the control point.

### Claim 24

A non-transitory processor readable medium storing instructions that when executed by a processor, cause the processor to:
receive, from a sensor, a detection of a first control point to determine a first spatial search of a first inventory;
detect, by a machine learning model and based on depth calculation of the first spatial search, a first plurality of storage units to calculate a storage unit count from a first plurality of storage unit counts, each storage unit count from the first plurality of storage unit counts is associated with a unit type from a plurality of unit types;
generate a digital model from a first of plurality digital models and that is overlayed around each storage unit from the first plurality of storage units;
store first inventory data in a database, such that the first plurality of digital models is hidden;
receive, from the sensor, a detection of a second control point to determine a second spatial search of a second inventory;
detect, by the machine learning model and based on depth calculation of the second spatial search, a second plurality of storage units to calculate a storage unit count from a second plurality of storage unit counts, each storage unit count from the second plurality of storage unit counts is associated with a unit type from the plurality of unit types; and
generate a digital model from a second plurality of digital models and that is overlayed around each storage unit from the second plurality of storage units.

### Claim 25 (depends on 24)

The non-transitory processor readable medium of claim 24, wherein:
the first inventory data includes metadata of the first plurality of storage units, the first plurality of storage unit counts, and the first plurality of digital models, and
second inventory data includes metadata of the second plurality of storage units, the second plurality of storage unit counts, and the second plurality of digital models.

### Claim 26 (depends on 25)

The non-transitory processor readable medium of claim 25, wherein the processor is further caused to distinguish the first plurality of storage units and the second plurality of storage units having a same unit type.