# US 11,907,899 B2 · Counting a number of objects in an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,899 B2
- Document ID: US11907899B2
- Status: Granted
- Kind: B2
- Application number: 17/410,849
- Filed: 2021-08-24
- Published: 2022-04-14
- Granted: 2024-02-20
- Earliest claimed: 2020-10-12
- Assignees: Insight Direct USA, Inc.
- Applicants: Insight Direct USA, Inc.
- Inventors: Michael Griffin
- Web page: https://patentgrep.com/patent/US11907899B2

## CPC

- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06V 20/64: Three-dimensional [3D] objects
- G06Q 10/08772: {Inventory counting methods}
- G06T 11/20: Drawing from basic elements
- G06V 10/457: {by analysing connectivity, e.g. edge linking, connected component analysis or slices}
- G06V 20/20: in augmented reality scenes
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06T 2210/12: Bounding box
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A product count of a number of physical products within a physical grouping of a plurality of the physical products is determined based on an image of the physical grouping. A cycle counter generates image coordinates for visible product surfaces within the image. The cycle counter generates three-dimensional virtual base coordinates for the expected locations of the plurality of the physical products within the physical grouping. The cycle counter determines the actual locations of the visible product surfaces within three-dimensional space based on a comparison of the image coordinates and the virtual base coordinates. The cycle counter determines the product count of the number of physical products within the physical grouping based on the actual locations.

## Claims

### Claim 1

A method of cycle counting, the method comprising:
receiving, by a computing device, captured image data of a plurality of physical products forming a physical grouping;
identifying, by the computing device, visible surfaces of the plurality of physical products based on the captured image data and by a recognition computer vision model trained on baseline image data corresponding to physical parameters of the plurality of physical products, to generate identified visible surfaces;
generating, by the computing device, image coordinates for the identified visible surfaces, wherein each image coordinate is a two-dimensional coordinate associated with one of the identified visible surfaces;
generating, by the computing device, a virtual product grouping and base coordinates corresponding to a plurality of virtual products forming the virtual product grouping, wherein each base coordinate is a three-dimensional coordinate associated with a virtual product of the plurality of virtual products;
associating, by the computing device, each identified visible surface with a virtual surface of the plurality of virtual products based on a comparison of the image coordinates and the base coordinates;
classifying, by the computing device, each identified visible surface as horizontal or vertical, thereby generating classified visible surfaces;
determining, by the computing device, a product count of the plurality of physical products in the physical grouping based on an order of the classified visible surfaces; and
outputting, by the computing device, the product count.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying, by the computing device, the visible surfaces of the plurality of physical products includes applying, by the computing device, bounding boxes to the captured image data to identify each of the identified visible surfaces.

### Claim 3 (depends on 1)

The method of claim 1, wherein identifying, by the computing device, the visible surfaces of the plurality of physical products includes applying, by the computing device, pixel masks to the captured image data to identify each of the identified visible surfaces.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
identifying, by the computing device, a common location between a two-dimensional space of the captured image data and a three-dimensional space of the virtual product grouping; and
utilizing the common location as a coordinate origin for generating the image coordinates and the base coordinates.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
configuring, by the computing device, the captured image data based on a support surface of the physical product grouping that is visible within the captured image data to generate configured image data, the configured image data having a smaller image area than the captured image data.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
establishing, by the computing device, the coordinate origin based on the support surface.

### Claim 7 (depends on 6)

The method of claim 6, wherein establishing, by the computing device, the coordinate origin based on the support surface includes establishing the coordinate origin at a front corner of the support surface.

### Claim 8 (depends on 5)

The method of claim 5, wherein, configuring, by the computing device, the captured image data includes configuring the captured image data by a configuration computer vision model trained on baseline image data corresponding to physical parameters of the physical grouping.

### Claim 9 (depends on 1)

The method of claim 1, wherein classifying, by the computing device, each identified visible surface includes classifying each identified visible surface as horizontal or vertical by the recognition computer vision model.

### Claim 10 (depends on 1)

The method of claim 1, wherein classifying, by the computing device, each identified visible surface includes classifying each identified visible surface as horizontal or vertical based on the comparison between the image coordinates and the base coordinates.

### Claim 11 (depends on 1)

The method of claim 1, wherein determining, by the computing device, the product count of the plurality of physical products in the physical grouping based on the order of the identified visible surfaces includes:
determining, by the computing device, a first column count for a first row of the plurality of physical products in the physical grouping based on a first count of a number of horizontal portions present in the first row; and
determining by the computing device, a first layer count for the first row of the plurality of physical products in the physical group based on a second count of a number of vertical portions present in the first row.

### Claim 12 (depends on 1)

The method of claim 1, wherein generating, by the computing device, the image coordinates for the identified visible surfaces includes:
locating, by the computing device, the image coordinates at centroids of the identified visible surfaces.

### Claim 13 (depends on 12)

The method of claim 12, wherein generating, by the computing device, the virtual product grouping and the base coordinates corresponding to the plurality of virtual products forming the virtual product grouping further includes:
determining, by the computing device, an expected location of each physical product of the plurality of physical products forming the physical grouping relative to each other physical product of the plurality of physical products forming the physical grouping; and
locating, by the computing device, the base coordinates within the three-dimensional space of the virtual product grouping based on the expected location of each physical product of the plurality of physical products.

### Claim 14 (depends on 13)

The method of claim 13, wherein generating, by the computing device, the virtual product grouping and assigning the base coordinates to the virtual product grouping further includes:
generating, by the computing device, the virtual product grouping based on the expected location of each of the plurality of physical products within the physical grouping and based on product information regarding the plurality of physical products;
assigning, by the computing device, the base coordinates to virtual surfaces of the plurality of virtual products.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
locating, by the computing device, the base coordinates at centroids of the virtual surfaces.

### Claim 16 (depends on 1)

The method of claim 1, further comprising:
receiving, by the computing device, product information regarding at least one physical product of the plurality of physical products forming the physical grouping, the product information including dimensions of the at least one physical product.

### Claim 17 (depends on 16)

The method of claim 16, wherein generating, by the computing device, the virtual product grouping and base coordinates corresponding to the plurality of virtual products forming the virtual product grouping includes:
generating, by the computing device, the virtual product grouping based on the product information.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
classifying, by the computing device, each identified visible surface as horizontal or vertical, thereby generating classified visible surfaces before associating, by the computing device, each identified visible surface with the virtual surface of the plurality of virtual products based on the comparison of the image coordinates and the base coordinates such that the comparison of the image coordinates and the base coordinates is conducted based on classified visible surfaces.

### Claim 19 (depends on 1)

The method of claim 1, wherein outputting, by the computing device, the product count includes outputting the product count to an inventory tracking system.

### Claim 20

A cycle counter configured to generate product counts for a physical grouping formed by a plurality of physical product, the cycle counter comprising:
control circuitry; and
memory encoded with instructions that, when executed by the control circuitry, cause the control circuitry to:
identify visible surfaces of the plurality of physical products based on captured image data received by an image-captured device and by a recognition computer vision model trained on baseline image data corresponding to physical parameters of the plurality of physical products, to generate identified visible surfaces;
generate image coordinates for the identified visible surfaces, wherein each image coordinate is a two-dimensional coordinate associated with one of the identified visible surfaces;
generate a virtual product grouping and base coordinates corresponding to a plurality of virtual products forming the virtual product grouping, wherein each base coordinate is a three-dimensional coordinate associated with a virtual product of the plurality of virtual products;
associate each identified visible surface with a virtual surface of the plurality of virtual products based on a comparison of the image coordinates and the base coordinates;
classify each identified visible surface as horizontal or vertical, thereby generating classified visible surfaces;
determine a product count of the plurality of physical products in the physical grouping based on an order of the classified visible surfaces; and
output the product count.