# US 2021/0334590 A1 · Automated image curation for machine learning deployments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334590 A1
- Document ID: US20210334590A1
- Status: Published application
- Kind: A1
- Application number: 16/859,244
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: Toshiba Global Commerce Solutions Holdings Corporation
- Applicants: Toshiba Global Commerce Solutions Holdings Corporation
- Inventors: Adrian RODRIGUEZ; Brian C. YOUNG; Bruno ROY; Jonathan WAITE
- Web page: https://patentgrep.com/patent/US20210334590A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure provides techniques for data curation and image evaluation. A first image is captured, and a first indication of a first item is received. A first identifier of the first item is then identified based on the first indication. Further, based on the first indication, it is determined that the first image depicts the first item. The first image is labeled with the first identifier, and a machine learning (ML) model of an ML system is trained based on the labeled first image.

## Claims

### Claim 1

A method, comprising:
capturing a first image;
receiving a first indication of a first item;
identifying, based on the first indication, a first identifier of the first item;
determining, based on the first indication, that the first image depicts the first item;
labeling the first image with the first identifier; and
training a machine learning (ML) model of an ML system based on the labeled first image.

### Claim 2 (depends on 1)

The method of claim 1, wherein training the ML model based on the labeled first image is performed based on determining that the ML system is in a training phase.

### Claim 3 (depends on 1)

The method of claim 1, the method further comprising:
capturing a second image; and
generating a predicted identifier based on processing the second image with the ML model.

### Claim 4 (depends on 3)

The method of claim 3, the method further comprising:
determining that the ML system is in a verification state;
receiving a second indication of the first item;
identifying, based on the second indication, the first identifier of the first item; and
storing the second image, along with the predicted identifier and the first identifier.

### Claim 5 (depends on 3)

The method of claim 3, the method further comprising:
determining that the predicted identifier and the first identifier do not match; and
refining the ML model based on the second image and the first identifier.

### Claim 6 (depends on 3)

The method of claim 3, the method further comprising:
determining that the ML system is in an active state;
refraining from receiving an indication of the first item; and
logging the predicted identifier.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining, based on the first indication, that the first image depicts the first item comprises:
identifying a timestamp associated with the first indication;
identifying a location associated with the first indication; and
determining that the timestamp and location associated with the first indication align with a timestamp and location associated with the first image.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining, based on the first indication, that the first image depicts the first item comprises:
identifying a location associated with the first indication; and
in response to receiving the first indication, causing the first image to be captured of the location associated with the first indication.

### Claim 9

A computer-readable storage medium containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising:
capturing a first image;
receiving a first indication of a first item;
identifying, based on the first indication, a first identifier of the first item;
determining, based on the first indication, that the first image depicts the first item;
labeling the first image with the first identifier; and
training a machine learning (ML) model of an ML system based on the labeled first image.

### Claim 10 (depends on 9)

The computer-readable storage medium of claim 9, the operation further comprising:
capturing a second image; and
generating a predicted identifier based on processing the second image with the ML model.

### Claim 11 (depends on 10)

The computer-readable storage medium of claim 10, the operation further comprising:
determining that the ML system is in a verification state;
receiving a second indication of the first item;
identifying, based on the second indication, the first identifier of the first item; and
storing the second image, along with the predicted identifier and the first identifier.

### Claim 12 (depends on 10)

The computer-readable storage medium of claim 10, the operation further comprising:
determining that the predicted identifier and the first identifier do not match; and
refining the ML model based on the second image and the first identifier.

### Claim 13 (depends on 10)

The computer-readable storage medium of claim 10, the operation further comprising:
determining that the ML system is in an active state;
refraining from receiving an indication of the first item; and
logging the predicted identifier.

### Claim 14 (depends on 9)

The computer-readable storage medium of claim 9, wherein determining, based on the first indication, that the first image depicts the first item comprises:
identifying a timestamp associated with the first indication;
identifying a location associated with the first indication; and
determining that the timestamp and location associated with the first indication align with a timestamp and location associated with the first image.

### Claim 15

A system comprising:
one or more computer processors; and
a memory containing a program which when executed by the one or more computer processors performs an operation, the operation comprising:
capturing a first image;
receiving a first indication of a first item;
identifying, based on the first indication, a first identifier of the first item;
determining, based on the first indication, that the first image depicts the first item;
labeling the first image with the first identifier; and
training a machine learning (ML) model of an ML system based on the labeled first image.

### Claim 16 (depends on 15)

The system of claim 15, the operation further comprising:
capturing a second image; and
generating a predicted identifier based on processing the second image with the ML model.

### Claim 17 (depends on 16)

The system of claim 16, the operation further comprising:
determining that the ML system is in a verification state;
receiving a second indication of the first item;
identifying, based on the second indication, the first identifier of the first item; and
storing the second image, along with the predicted identifier and the first identifier.

### Claim 18 (depends on 16)

The system of claim 16, the operation further comprising:
determining that the predicted identifier and the first identifier do not match; and
refining the ML model based on the second image and the first identifier.

### Claim 19 (depends on 16)

The system of claim 16, the operation further comprising:
determining that the ML system is in an active state;
refraining from receiving an indication of the first item; and
logging the predicted identifier.

### Claim 20 (depends on 15)

The system of claim 15, wherein determining, based on the first indication, that the first image depicts the first item comprises:
identifying a timestamp associated with the first indication;
identifying a location associated with the first indication; and
determining that the timestamp and location associated with the first indication align with a timestamp and location associated with the first image.
