# US 12,118,506 B2 · System and method for associating products and product labels > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,506 B2 - Document ID: US12118506B2 - Status: Granted - Kind: B2 - Application number: 17/425,290 - Filed: 2020-04-13 - Published: 2022-05-05 - Granted: 2024-10-15 - Earliest claimed: 2019-04-11 - Assignees: Carnegie Mellon University; Bossa Nova Robotics - Applicants: Carnegie Mellon University; Bossa Nova Robotics - Inventors: Sarjoun Skaff; Marios Savvides; Uzair Ahmed; Sreena Nallamothu; Ran Tao; Nikhil Mohan - Web page: https://patentgrep.com/patent/US12118506B2 ## CPC - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 10/08724: {by shelf level inventory management, e.g. planograms} - B65G 1/137: with arrangements or automatic control means for selecting which articles are to be removed - G06K 7/1413: {1D bar codes} - G06K 7/1417: {2D bar codes} - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 7/60: Analysis of geometric attributes - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/16: {using multiple overlapping images; Image stitching} - G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition - G06V 10/225: {based on a marking or identifier characterising the area} - G06V 10/764: using classification, e.g. of video objects - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G09F 3/0297: {including a machine-readable marking, e.g. a bar code (digitally marked record carriers in general G06K19/00)} - G09F 3/204: {specially adapted to be attached to a shelf or the like} - H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - G06T 2207/20081: Training; Learning - G06T 2207/30232: Surveillance - G06V 2201/07: Target detection - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D) - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09F: DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An automated inventory monitoring system includes an image capture module able to create an image of an aisle of a retail store. Images of products labels are identified in the image and classified as shelf labels or peg labels. For shelf labels, an area of the shelf is defined and associated with the shelf label. Images of products are identified in the image and products on the shelf within an area associated with a shelf label are associated with the shelf label. Products located below a peg label are associated with the peg label. Based on the association between labels and products, out-of-stock products, plugs and spread may be detected and reported to the staff of the retail store. ## Claims ### Claim 1 A system comprising: one or more processors; one or more cameras coupled to the one or more processors; and software implementing: a product detection neural network; a label detection neural network; and a shelf detection neural network; wherein the software causes the system to: obtain a representation of an aisle containing stocked products, the representation comprising one or more images of the aisle captured by the one or more cameras; input the one or more images to the product detection neural network to obtain bounding boxes of products detected in the one or more images; input the one or more images to the label detection neural network to obtain bounding boxes of labels detected in the one or more images; input the one or more images to the shelf detection neural network to obtain bounding boxes of shelves detected in the one or more images; determine that one or more product labels are shelf labels by determining that the bounding boxes for the product labels overlap a bounding box for a shelf; determine an area associated with each shelf label, defined as an area on a shelf defined by the shelf label and an adjacent shelf label; and associate products having bounding boxes substantially overlapping the areas associated with a shelf label with that shelf label. ### Claim 2 (depends on 1) The system of claim 1, further comprising: establishing a coordinate system superimposed on the representation of the aisle; determining the positions of the product label and the product bounding boxes of the products, the label and the shelves with respect to the coordinate system. ### Claim 3 (depends on 2) The system of claim 2, the bounding boxes for the products, the labels and the shelves being identified by tuples comprising at least the coordinates within the coordinate system of the bouding boxes. ### Claim 4 (depends on 3) The system of claim 3 wherein identifying a product label in the representation of the aisle comprises: submitting the label detected by the label detection neural network to a label classifier to determine if the label is a product label or another type of label; and adding the tuple to a list of product labels if the image of the label is classified as a product label. ### Claim 5 (depends on 2) The system of claim 2, wherein identifying a product in the representation of the aisle comprises: determining the type of the product using a product classifier; and adding the tuple and a type of the product to a list of products. ### Claim 6 (depends on 1) The system of claim 1, further comprising: determining that one or more product labels are peg labels by determining that the bounding box for the product labels do not overlap a bounding box for a shelf. ### Claim 7 (depends on 6) The system of claim 6 wherein defining the area associated with the peg label comprises: defining an area directly below the bounding box for the peg label and areas adjacent to the area directly below the bounding box for the peg label having no associated peg labels as the area associated with the peg label. ### Claim 8 (depends on 7) The system of claim 7 wherein determining that the product lies within the area associated with the peg label comprises: defining that the bounding box for the product overlaps at least partially with the area associated with the peg label. ### Claim 9 (depends on 1) The system of claim 1 wherein the area associated with a shelf label is defined as an area above or below a shelf between an offset from a vertical coordinate for the shelf label and an offset from a corresponding vertical coordinate for an adjacent shelf label. ### Claim 10 (depends on 1) The system of claim 1, further comprising determining that a product lies within the area associated with the shelf label by determining that the width of the bounding box for the product overlaps the area associated with the shelf label by a predetermined percentage. ### Claim 11 (depends on 10) The system of claim 10 wherein determining the overlap between the width of the bounding box for the product and the area associated with a shelf label comprises dividing the distance between the leftmost coordinate of the area associated with the shelf label and the rightmost coordinate of the product bounding box by the overall width of the product bounding box. ### Claim 12 (depends on 1) The system of claim 1, further comprising: determining that two products match or do not match. ### Claim 13 (depends on 12) The system of claim 12, further comprising: determining that two adjacent products on the same shelf do not match; determining that the two adjacent products are within the area associated with the same shelf label; and flagging the detection of a plug. ### Claim 14 (depends on 12) The system of claim 12, further comprising: determining that two adjacent products on the same shelf match; determining that the two adjacent products are not within the area associated with the same shelf label; and flagging the detection of a spread. ### Claim 15 (depends on 12) The system of claim 12 wherein determining that two products match or do not match comprises: determining if sizes of the bounding boxes associated with the adjacent products match within a predetermined threshold and if not, determining that the adjacent products do not match; determining that color distributions of the products within the bounding boxes match within a predetermined threshold and, if not, determining that the adjacent products do not match; and submitting adjacent products to a trained model to perform deep feature matching between the products, the trained model providing an indication that the products match or do not match. ### Claim 16 (depends on 12) The system of claim 12 wherein determining that two products match or do not match comprises: submitting each product to a product identification classifier; receiving as output from the product identification classifier an identification of each product; and determining that the identifications of the products match or do not match. ### Claim 17 (depends on 1) The system of claim 1 wherein obtaining an image a representation of an aisle comprises: capturing images of sections of the aisle; and stitching images for two or more sections together to create a stitched image. ### Claim 18 (depends on 17) The system of claim 17, the stitched image being a panoramic image showing the entirety of the aisle.