# US 2021/0280091 A1 · System and method for building machine learning or deep learning data sets for recognizing labels on items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280091 A1
- Document ID: US20210280091A1
- Status: Published application
- Kind: A1
- Application number: 17/191,483
- Filed: 2021-03-03
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: United States Postal Service
- Applicants: United States Postal Service
- Inventors: Ryan J. Simpson
- Web page: https://patentgrep.com/patent/US20210280091A1

## CPC

- B65C 1/02: Affixing labels to one flat surface of articles, e.g. of packages, of flat bands
- B25J 11/00: Manipulators not otherwise provided for
- G06K 7/00: Methods or arrangements for sensing record carriers, {e.g. for reading patterns} (methods or arrangements for marking the record carrier in digital fashion G06K1/00; pattern recognition G06F18/00; arrangements for image or video recognition or understanding G06V10/00; character recognition, recognising digital ink or document-oriented image-based pattern recognition G06V30/00)
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 50/26: Government or public services (business processes related to the transportation industry G06Q50/40)
- G09F 3/0288: {Labels or tickets consisting of more than one part, e.g. with address of sender or other reference on separate section to main label; Multi-copy labels}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N 20/20: Ensemble learning
- B65C: LABELLING OR TAGGING MACHINES, APPARATUS, OR PROCESSES (nailing or stapling in general B25C, B27F; applying labels for packaging purposes B65B; labels, name-plates G09F ){(series-printing machines B41L; containers for the storage or transport of packages B65D)}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G09F: DISPLAYING; ADVERTISING; SIGNS; LABELS OR NAME-PLATES; SEALS

## Abstract

This application relates to a method and a system for building machine learning or deep learning data sets for automatically recognizing labels on items. The system may include an optical scanner configured to capture an item including one or more labels provided thereon, the item captured a plurality of times at different positions with respect to the optical scanner. The system may further include a robotic arm on which the item is disposed, the robotic arm configured to rotate the item horizontally and/or vertically such that the one or more labels of the item are captured by the optical scanner at different positions with respect to the optical scanner. The system may include a database configured to store the captured images.

## Claims

### Claim 1

A system for building machine learning or deep learning data sets for automatically recognizing information on items, the system comprising:
an optical scanner configured to capture an item including one or more pieces of information provided thereon, the item captured a plurality of times at different positions with respect to the optical scanner;
a robotic arm on which the item is disposed, the robotic arm configured to rotate the item horizontally and/or vertically such that the one or more pieces of the information of the item are captured by the optical sensor at the different positions with respect to the optical scanner; and
a database configured to store the captured images of the one or more pieces of information of the item.

### Claim 2 (depends on 1)

The system of claim 1, wherein the item comprises a mail item, and wherein the one or more pieces of the information comprise an address section, a sender section, a recipient section, a barcode section, a postage section, a special item section and a label.

### Claim 3 (depends on 2)

The system of claim 2, wherein the label comprises a service class indicator label, a warning label or a hazardous label.

### Claim 4 (depends on 3)

The system of claim 3, wherein the hazardous label comprises a department of transportation (DOT) hazard class label.

### Claim 5 (depends on 1)

The system of claim 1, wherein the controller is further configured to move at least one of the optical scanner and the robotic arm to at least partially align a lens of the optical scanner with the item disposed on the robotic arm.

### Claim 6 (depends on 1)

The system of claim 1, further comprising a light source configured to emit light to the one or more pieces of information provided on the item.

### Claim 7 (depends on 6)

The system of claim 6, wherein the controller is further configured to control one or more of the optical scanner, the robotic arm and the light source such that the one or more pieces of information provided on the item are captured by the optical scanner in different capturing environments.

### Claim 8 (depends on 7)

The system of claim 7, wherein the different capturing environments comprise one or more of robotic arm or item movement speeds, optical scanner shutter speeds, focus, intensity, color, white balance, lightings, shadows, contrasts, lighting conditions, or combinations thereof.

### Claim 9 (depends on 1)

The system of claim 1, wherein the positions comprise one or more of angles or distances between a lens of the optical scanner and the item, or orientations of the item with respect to the lens of the optical scanner.

### Claim 10 (depends on 1)

The system of claim 1, further comprising:
a plate to which the item is attached; and
rotation mechanism provided on the robotic arm and configured to rotate the plate.

### Claim 11

A method of building machine learning or deep learning data sets for automatically recognizing information on items, the method comprising:
capturing, at an optical scanner, an item including one or more pieces of information provided thereon, the item captured a plurality of times at different positions with respect to the optical scanner;
rotating the item, via a robotic arm on which the item is disposed, horizontally and/or vertically such that the one or more pieces of the information of the item are captured by the optical scanner at the different positions with respect to the optical scanner; and
storing, at a database, the captured images of the one or more pieces of information of the item.

### Claim 12 (depends on 11)

The method of claim 11, wherein the storing comprises storing relationships between the different positions of the one or more pieces of the information of the item and whether the captured one or more pieces of the information of the item are fully recognized by the optical scanner.

### Claim 13 (depends on 11)

The method of claim 11, wherein the item comprises a mail item, and wherein the one or more pieces of the information comprise an address section, a sender section, a recipient section, a barcode section, a postage section, a special item section, and a label.

### Claim 14 (depends on 13)

The method of claim 13, wherein the label comprises a warning label, a department of transportation (DOT) hazard class label or a service classification indicator label.

### Claim 15 (depends on 11)

The method of claim 11, further comprising controlling, at a controller, the database to move a first portion of the captured images stored in the database to a training data DB and a second portion of the captured images stored in the database different from the first portion to a test data DB, the controller configured to train a machine learning or deep learning model for automatically recognizing labels on items with the first portion of the captured images stored in the training data DB, and test the trained machine learning or deep learning model with the second portion of the captured images stored in the test data DB.

### Claim 16 (depends on 15)

The method of claim 15, wherein the controlling comprises moving at least one of the optical scanner and the robotic arm to at least partially align a lens of the optical scanner with the item disposed on the robotic arm.

### Claim 17 (depends on 11)

The method of claim 11, further comprising emitting, at a light source, light to the one or more pieces of the information of the item.

### Claim 18 (depends on 17)

The method of claim 17, further comprising controlling, at the controller, one or more of the optical scanner, the robotic arm and the light source such that the one or more pieces of the information of the item are captured by the optical scanner in different capturing environments.

### Claim 19 (depends on 18)

The method of claim 18, wherein the different capturing environments comprise robotic arm or item movement speeds, optical scanner shutter speeds, focus, intensity, color, white balance, lightings, shadows, contrasts, lighting conditions or combinations thereof.

### Claim 20 (depends on 11)

The method of claim 11, wherein the positions comprise one or more of angles or distances between a lens of the optical scanner and the one or more pieces of the information of the item, orientations of the one or more pieces of the information of the item with respect to the lens of the optical scanner.