# US 2021/0081707 A1 · System for scaling fine-grained object recognition of consumer packaged goods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081707 A1
- Document ID: US20210081707A1
- Status: Published application
- Kind: A1
- Application number: 17/106,091
- Filed: 2020-11-28
- Published: 2021-03-18
- Earliest claimed: 2018-06-17
- Assignees: Pensa Systems, Inc.
- Applicants: Pensa Systems, Inc.
- Inventors: Joel Iventosch; James E. Dutton; Sean D. Matthews
- Web page: https://patentgrep.com/patent/US20210081707A1

## CPC

- G06Q 10/06315: {Needs-based resource requirements planning or analysis}
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/2431: Multiple classes
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06T 7/73: using feature-based methods
- G06T 11/20: Drawing from basic elements
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/176: {Urban or other man-made structures}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2210/12: Bounding box
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system is provided for identifying consumer packaged goods (CPGs). The system comprises an imaging device mounted on a mobile platform; a CPG detector which is equipped with a neural network and which (a) utilizes the imaging device to obtain an image containing a plurality of CPGs, (b) detects a set of CPG superclass features in the image, and (c) generates a CPG feature map which maps the location of CPG features in the image, wherein said neural network has been trained on a set of images of CPGs to recognize which of a set of superclasses a detected CPG belongs to; a region proposal network which accepts the CPG feature map as input and which returns a first set of regions in the image where a CPG could be located; a superclassifier which examines each region in the first set of regions to determine if the region contains an instance of a superclass of a CPG, and which outputs a second set of regions containing a CPG; and a superclass proposal layer which operates on each member of the second set of regions and returns a set of superclass proposals for each region.

## Claims

### Claim at position 1

1-27. (canceled)

### Claim 28

A system for identifying consumer packaged goods (CPGs), comprising:
an imaging device mounted on a mobile platform;
a CPG detector which is equipped with a neural network and which
(a) utilizes the imaging device to obtain an image containing a plurality of CPGs,
(b) detects a set of CPG superclass features in the image, and
(c) generates a CPG feature map which maps the location of CPG features in the image,
wherein said neural network has been trained on a set of images of CPGs to recognize which of a set of superclasses a detected CPG belongs to;
a region proposal network which accepts the CPG feature map as input and which returns a first set of regions in the image where a CPG could be located;
a superclassifier which examines each region in the first set of regions to determine if the region contains an instance of a superclass of a CPG, and which outputs a second set of regions containing a CPG; and
a superclass proposal layer which operates on each member of the second set of regions and returns a set of superclass proposals for each region.

### Claim 29 (depends on 28)

The system of claim 28, wherein said neural network is a deep convolutional neural network (CNN).

### Claim 30 (depends on 29)

The system of claim 29, wherein said CNN is equipped with a convolutional layer having features recognized in the image.

### Claim 31 (depends on 28)

The system of claim 28, wherein said proposal network finds all possible locations in the image where a CPG could be located.

### Claim 32 (depends on 28)

The system of claim 28, wherein the set of regions is combined using a region of interest pooling (RoI) technique.

### Claim 33 (depends on 32)

The system of claim 32, wherein said RoI technique uses a neural-net layer of the neural network for object detection.

### Claim 34 (depends on 28)

The system of claim 28, wherein the superclassifier implements a Faster R-CNN technique.

### Claim 35 (depends on 28)

The system of claim 28, wherein said superclass proposal layer operates on each member of the second set of regions and returns a plurality of superclass proposals for each region.

### Claim 36 (depends on 28)

The system of claim 28, wherein said superclass proposal layer operates on each member of the second set of regions with a normalized exponential function to generate the set of superclass proposals for each region.

### Claim 37 (depends on 28)

The system of claim 28, wherein each superclass proposal in the set of superclass proposals is a trinomial variable whose values are CPG superclass, confidence score and bounding box.

### Claim 38 (depends on 35)

The system of claim 35, further comprising:
a plurality of subclassifiers which input the superclass proposals and output CPGs which fall within a superclass category.

### Claim 39 (depends on 38)

The system of claim 38, wherein each superclass has a sub-classifier associated with it which identifies CPGs that fall within that superclass.

### Claim 40 (depends on 39)

The system of claim 39, wherein the set of superclass proposals contains subsets B=[B1, . . . , Bk], wherein k≥1, wherein each subset Bn is a set of superclass proposals which identifies superclass Sn as the corresponding superclass, wherein subclassifier Cn identifies CPG classes that fall within superclass Sn, and wherein subclassifer Cn operates on subset Bn to return a CPG class which corresponds to each proposal in subset Bn.

### Claim 41 (depends on 39)

The system of claim 39, wherein the set of superclass proposals contains subsets B=[B1, . . . , Bk], wherein k≥1, wherein each subset Bn is a set of superclass proposals which identifies superclass Sn as the corresponding superclass, wherein subclassifier Cn identifies CPG classes that fall within superclass Sn, wherein subclassifer Cn operates on subset Bn to return a CPG class which corresponds to each proposal in subset Bn and a confidence score, and wherein the confidence score reflects the certainty with which the CPG was identified.

### Claim 42 (depends on 41)

The system of claim 41, wherein the CPG class returned by subclassifier Cn maps to a single SKU or UPC.

### Claim 43 (depends on 28)

The system of claim 28, wherein said imaging device is mounted on a drone.

### Claim 44 (depends on 43)

The system of claim 43, wherein said region proposal network, said superclassifier, and said superclass proposal layer are defined in an operating system running in a tangible, non-transient memory medium installed on said drone.

### Claim 45 (depends on 43)

The system of claim 43, wherein said region proposal network, said superclassifier, and said superclass proposal layer are defined in an operating system running in a tangible, non-transient memory medium installed in a computer system that is in communication with said drone.
