# US 2022/0027599 A1 · Automated training data collection for object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0027599 A1
- Document ID: US20220027599A1
- Status: Published application
- Kind: A1
- Application number: 16/939,698
- Filed: 2020-07-27
- Published: 2022-01-27
- Earliest claimed: 2020-07-27
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Florian Pinel; Russell Patrick Bobbitt; Sharon Laquinta Cham
- Web page: https://patentgrep.com/patent/US20220027599A1

## CPC

- G06V 20/64: Three-dimensional [3D] objects
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/20: Drawing from basic elements
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/87: {using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system}
- G06V 20/36: {Indoor scenes}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T 2210/12: Bounding box
- G06V 2201/10: Recognition assisted with metadata
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method, system, and computer program product for automated collection of training data and training object detection models is provided. The method generates a set of reference images for a first set of products. Based on the set of reference images, the method identifies a subset of products within an image stream. Based on the subset of products, a second set of products is determined within the image stream. The method identifies a set of product gaps based on the subset of products and the second set of products. The method generates a product detection model based on the set of reference images, the subset of products, the second set of products, and the product gaps.

## Claims

### Claim 1

A computer-implemented method, comprising:
generating a set of reference images for a first set of products;
based on the set of reference images, identifying a subset of products within an image stream;
based on the subset of products, determining a second set of products within the image stream;
based on the subset of products and the second set of products, identifying a set of product gaps; and
generating a product detection model based on the set of reference images, the subset of products, the second set of products, and the product gaps.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the set of reference images further comprises:
detecting an identifier for each product of the first set of products;
based on detecting the identifier, capturing one or more images of each product; and
associating each image with the identifier for a product depicted within the image.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein determining the second set of products further comprises:
identifying a set of expected products associated with one or more products of the subset of products; and
determining a subset of expected products within the image stream, the subset of expected products being the second set of products.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein identifying the set of product gaps further comprises:
generating a set of bounding boxes around the subset of products and the second set of products;
detecting one or more unbounded areas within a field of view of the image stream;
extracting a set of candidate gaps from the one or more unbounded areas; and
excluding a subset of candidate gaps.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a set of bounding boxes for the subset of products and the second set of products;
identifying one or more regions outside of the set of bounding boxes; and
generating an area of interest within a field of view of the image stream including the set of bounding boxes and excluding at least a portion of the one or more regions.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the product detection model is generated as a first product detection model and a second product detection model, the first product detection model generated for a first image capture device and the second product detection model generated for a second image capture device, and wherein the method further comprises:
capturing, at the first image capture device, a first image of products within a field of view of the first image capture device;
capturing, at the second image capture device, a second image of products within a field of view of the second image capture device;
detecting, using the first product detection model, a third set of products within the field of view of the first image capture device;
detecting, using the second product detection model, a fourth set of products within the field of view of the second image capture device; and
determining, based on the first set of products, one or more products missing from one or more of the third set of products and the fourth set of products.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the product detection model is generated as a plurality of zone-specific detection models with each zone-specific detection model being generated for a zone selected from a plurality of zones, and wherein the method further comprises:
capturing a first image from a mobile image capture device traversing the plurality of zones;
identifying a first zone of the plurality of zones, the mobile image capture device being in the first zone while capturing the first image;
identifying a first zone-specific detection model for the first image based on the first zone; and
based on the first set of products, the first image, and the first zone-specific detection model, determining one or more missing products for the first zone.

### Claim 8

A system, comprising:
one or more processors; and
a computer-readable storage medium, coupled to the one or more processors, storing program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
generating a set of reference images for a first set of products;
based on the set of reference images, identifying a subset of products within an image stream;
based on the subset of products, determining a second set of products within the image stream;
based on the subset of products and the second set of products, identifying a set of product gaps; and
generating a product detection model based on the set of reference images, the subset of products, the second set of products, and the product gaps.

### Claim 9 (depends on 8)

The system of claim 8, wherein generating the set of reference images further comprises:
detecting an identifier for each product of the first set of products;
based on detecting the identifier, capturing one or more images of each product; and
associating each image with the identifier for a product depicted within the image.

### Claim 10 (depends on 8)

The system of claim 8, wherein determining the second set of products further comprises:
identifying a set of expected products associated with one or more products of the subset of products; and
determining a subset of expected products within the image stream, the subset of expected products being the second set of products.

### Claim 11 (depends on 8)

The system of claim 8, wherein identifying the set of product gaps further comprises:
generating a set of bounding boxes around the subset of products and the second set of products;
detecting one or more unbounded areas within a field of view of the image stream;
extracting a set of candidate gaps from the one or more unbounded areas; and
excluding a subset of candidate gaps.

### Claim 12 (depends on 8)

The system of claim 8, wherein the operations further comprise:
generating a set of bounding boxes for the subset of products and the second set of products;
identifying one or more regions outside of the set of bounding boxes; and
generating an area of interest within a field of view of the image stream including the set of bounding boxes and excluding at least a portion of the one or more regions.

### Claim 13 (depends on 8)

The system of claim 8, wherein the product detection model is generated as a first product detection model and a second product detection model, the first product detection model generated for a first image capture device and the second product detection model generated for a second image capture device, and wherein the operations further comprise:
capturing, at the first image capture device, a first image of products within a field of view of the first image capture device;
capturing, at the second image capture device, a second image of products within a field of view of the second image capture device;
detecting, using the first product detection model, a third set of products within the field of view of the first image capture device;
detecting, using the second product detection model, a fourth set of products within the field of view of the second image capture device; and
determining, based on the first set of products, one or more products missing from one or more of the third set of products and the fourth set of products.

### Claim 14 (depends on 8)

The system of claim 8, wherein the product detection model is generated as a plurality of zone-specific detection models with each zone-specific detection model being generated for a zone selected from a plurality of zones, and wherein the operations further comprise:
capturing a first image from a mobile image capture device traversing the plurality of zones;
identifying a first zone of the plurality of zones, the mobile image capture device being in the first zone while capturing the first image;
identifying a first zone-specific detection model for the first image based on the first zone; and
based on the first set of products, the first image, and the first zone-specific detection model, determining one or more missing products for the first zone.

### Claim 15

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by one or more processors to cause the one or more processors to perform operations comprising:
generating a set of reference images for a first set of products;
based on the set of reference images, identifying a subset of products within an image stream;
based on the subset of products, determining a second set of products within the image stream;
based on the subset of products and the second set of products, identifying a set of product gaps; and
generating a product detection model based on the set of reference images, the subset of products, the second set of products, and the product gaps.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein generating the set of reference images further comprises:
detecting an identifier for each product of the first set of products;
based on detecting the identifier, capturing one or more images of each product; and
associating each image with the identifier for a product depicted within the image.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein determining the second set of products further comprises:
identifying a set of expected products associated with one or more products of the subset of products; and
determining a subset of expected products within the image stream, the subset of expected products being the second set of products.

### Claim 18 (depends on 15)

The computer program product of claim 15, wherein identifying the set of product gaps further comprises:
generating a set of bounding boxes around the subset of products and the second set of products;
detecting one or more unbounded areas within a field of view of the image stream;
extracting a set of candidate gaps from the one or more unbounded areas; and
excluding a subset of candidate gaps.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the product detection model is generated as a first product detection model and a second product detection model, the first product detection model generated for a first image capture device and the second product detection model generated for a second image capture device, and wherein the operations further comprise:
capturing, at the first image capture device, a first image of products within a field of view of the first image capture device;
capturing, at the second image capture device, a second image of products within a field of view of the second image capture device;
detecting, using the first product detection model, a third set of products within the field of view of the first image capture device;
detecting, using the second product detection model, a fourth set of products within the field of view of the second image capture device; and
determining, based on the first set of products, one or more products missing from one or more of the third set of products and the fourth set of products.

### Claim 20 (depends on 15)

The computer program product of claim 15, wherein the product detection model is generated as a plurality of zone-specific detection models with each zone-specific detection model being generated for a zone selected from a plurality of zones, and wherein the operations further comprise:
capturing a first image from a mobile image capture device traversing the plurality of zones;
identifying a first zone of the plurality of zones, the mobile image capture device being in the first zone while capturing the first image;
identifying a first zone-specific detection model for the first image based on the first zone; and
based on the first set of products, the first image, and the first zone-specific detection model, determining one or more missing products for the first zone.
