# US 2022/0277171 A1 · User-generated visual guide for the classification of images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277171 A1
- Document ID: US20220277171A1
- Status: Published application
- Kind: A1
- Application number: 17/500,784
- Filed: 2021-10-13
- Published: 2022-09-01
- Earliest claimed: 2019-09-27
- Assignees: Landing AI
- Applicants: Landing AI
- Inventors: Dongyan Wang; Gopi Prashanth Gopal; Andrew Yan-Tak Ng; Karthikeyan Thiruppathisamy Nathillvar; Rustam Hashimov; Pingyang He; Dillon Anthony Laird; Yiwen Rong; Alejandro Betancourt; Sanjeev Satheesh; Yu Qing Zhou
- Web page: https://patentgrep.com/patent/US20220277171A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/41: {Interactive pattern learning with a human teacher}
- G06N 5/04: Inference or reasoning models
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/945: {User interactive design; Environments; Toolboxes}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods are disclosed herein for creating a visual guide for developing training data for a classification of image, where the training data includes images tagged with labels for the classification of the images. A processor may prompt a user to define a framework for the classification. For an initial set of images within the training data, qualified human classifiers are prompted to locate the images within the framework and to tag the images with labels. The processor determines whether the tagged images have consistent labels, and, if so, the processor adds images to the training data. The processor may add the images by providing a visual guide, the visual guide including tagged images arranged according to their locations within the framework their labels, and prompting human classifiers to tag the additional images with labels for the classification, according to the visual guide.

## Claims

### Claim 1

A computer-implemented method for developing training data for a classification of images, the training data comprising images tagged with labels for the classification of the images, the method comprising:
prompting a user to define a framework for the classification, the framework, when defined, including example images that are each paired with one or more example labels;
prompting qualified human classifiers to tag an initial set of images within the training data with labels for the classification based on the example labels within the framework; and
adding the tagged images to the training data, wherein the training data is used to train a machine learning model to automatically tag further images with labels according to the framework.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising generating a visual guide comprising tagged images arranged according the framework.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, the visual guide comprising a prompt to tag additional images with labels according to the framework, the additional images, when tagged, being added to the training data.

### Claim 4 (depends on 1)

The method of claim 1, wherein adding the tagged images to the training data is performed responsive to determining that the tagged images have consistent labels.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining whether an image in the initial set is weakly classifiable; and
responsive to determining that the image is weakly classifiable, modifying that image to be strongly classifiable.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining whether the label for a tagged image in the initial set is incorrect; and
responsive to determining that the label is incorrect, correcting the label for that image prior to adding that image to the training data.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining whether the tagged images provide sufficient samples across the framework; and
responsive to determining that the tagged images do not provide sufficient samples, prompting to provide additional tagged images in regions of the framework that have insufficient samples.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the visual guide comprises a spectrum of tagged images spanning a range of values for the labels, the method further comprising automatically placing a new image on the spectrum, based on a location of the new image within the framework and a similarity of the new image relative to the tagged images spanning the spectrum.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising using the training data to benchmark an accuracy of human classifiers, wherein the human classifiers are benchmarked against a machine learning model trained using the training data.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the visual guide dynamically adapts by giving more weight to newer images compared to older images.

### Claim 11

A system comprising for developing training data for a classification of images, the training data comprising images tagged with labels for the classification of the images, the system comprising one or more processors that, when executing instructions, are caused perform operations comprising:
prompting a user to define a framework for the classification, the framework, when defined, including example images that are each paired with one or more example labels;
prompting qualified human classifiers to tag an initial set of images within the training data with labels for the classification based on the example labels within the framework; and
adding the tagged images to the training data, wherein the training data is used to train a machine learning model to automatically tag further images with labels according to the framework.

### Claim 12 (depends on 11)

The system of claim 11, the operations further comprising generating a visual guide comprising tagged images arranged according the framework.

### Claim 13 (depends on 12)

The system of claim 12, the visual guide comprising a prompt to tag additional images with labels according to the framework, the additional images, when tagged, being added to the training data.

### Claim 14 (depends on 11)

The system of claim 11, wherein adding the tagged images to the training data is performed responsive to determining that the tagged images have consistent labels.

### Claim 15 (depends on 11)

The system of claim 11, the operations further comprising:
determining whether an image in the initial set is weakly classifiable; and
responsive to determining that the image is weakly classifiable, modifying that image to be strongly classifiable.

### Claim 16 (depends on 11)

The system of claim 11, the operations further comprising:
determining whether the label for a tagged image in the initial set is incorrect; and
responsive to determining that the label is incorrect, correcting the label for that image prior to adding that image to the training data.

### Claim 17 (depends on 11)

The system of claim 11, the operations further comprising:
determining whether the tagged images provide sufficient samples across the framework; and
responsive to determining that the tagged images do not provide sufficient samples, prompting to provide additional tagged images in regions of the framework that have insufficient samples.

### Claim 18 (depends on 11)

The system of claim 11, wherein the visual guide comprises a spectrum of tagged images spanning a range of values for the labels, the operations further comprising automatically placing a new image on the spectrum, based on a location of the new image within the framework and a similarity of the new image relative to the tagged images spanning the spectrum.

### Claim 19 (depends on 11)

The system of claim 11, the operations further comprising using the training data to benchmark an accuracy of human classifiers, wherein the human classifiers are benchmarked against a machine learning model trained using the training data.

### Claim 20 (depends on 11)

The system of claim 11, wherein the visual guide dynamically adapts by giving more weight to newer images compared to older images.
