# US 11,907,336 B2 · Visual labeling for machine learning training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,336 B2
- Document ID: US11907336B2
- Status: Granted
- Kind: B2
- Application number: 17/516,948
- Filed: 2021-11-02
- Published: 2023-05-04
- Granted: 2024-02-20
- Earliest claimed: 2021-11-02
- Assignees: Sap SE
- Applicants: Sap SE
- Inventors: Ran M. Bittmann; Hans-Martin Ramsl
- Web page: https://patentgrep.com/patent/US11907336B2

## CPC

- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 30/10: Character recognition
- G06V 30/1914: {Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries, e.g. user dictionaries}
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06K 19/06028: {using bar codes}
- G06K 19/06037: {multi-dimensional coding}
- G06V 20/63: {Scene text, e.g. street names}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 30/19153: {using rules for classification or partitioning the feature space}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS

## Abstract

Systems, methods, and computer-readable media are disclosed for visual labeling of training data items for training a machine learning model. Training data items may be generated for training the machine learning model. Visual labels, such as QR codes, may be created for the training data items. The creation of the training data item and the visual label may be automated. The visual labels and the training data items may be combined to obtain a labeled training data item. The labeled training data item may comprise a separator to distinguish the training data item from the visual label. The labeled training data item may be used for training and validation of the machine learning model. The machine learning model may analyze the training data item, attempt to identify the training data item, and compare the identification against the embedded label.

## Claims

### Claim 1

One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, perform a method for machine learning training with embedded visual labels, the method comprising:
generating a plurality of training data items for training a machine learning model;
in response to generating the plurality of training data items, creating a plurality of visual labels, each visual label in the plurality of visual labels identifying a training data item of the plurality of training data items;
attaching each visual label in the plurality of visual labels to a training data item in the plurality of training data items to obtain a plurality of labeled training data items;
for each labeled training data item in the plurality of labeled training data items, defining a separator to separate the visual label from the training data item;
providing the plurality of labeled training data items to the machine learning model for training; and
for each labeled training data item in the plurality of labeled training data items:
receiving, from the machine learning model, a classification of the training data item; and
comparing the classification of the training data item against the visual label.

### Claim 2 (depends on 1)

The media of claim 1, wherein each visual label of the plurality of visual labels comprises a quick response (QR) code.

### Claim 3 (depends on 1)

The media of claim 1, wherein the machine learning model is an optical character recognition model.

### Claim 4 (depends on 1)

The media of claim 1, wherein the method further comprises:
altering the training data item within the labeled training data item,
wherein the altering comprises at least one of a rotation, a smudge, a strikethrough, a bolding, an italicizing, or a blurring of the training data item.

### Claim 5 (depends on 1)

The media of claim 1, wherein the method further comprises:
in response to determining that the classification is correct:
altering the training data item to obtain an altered labeled training data item;
receiving, from the machine learning model, a classification of the altered labeled training data item; and
comparing the classification of the altered labeled training data item to the visual label.

### Claim 6 (depends on 1)

The media of claim 1, wherein the method further comprises:
in response to determining that the classification is incorrect:
adjusting the machine learning model;
receiving, from the machine learning model, a reclassification of the training data item; and
comparing the reclassification of the training data item to the visual label.

### Claim 7 (depends on 1)

The media of claim 1,
wherein the separator comprises one of a solid line disposed between the training data item and the visual label or a dashed line disposed between the training data item and the visual label,
wherein the separator is disposed at a predefined location across the plurality of labeled training data items.

### Claim 8

A system for machine learning training with embedded visual labels, comprising:
a data store;
a processor; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the processor, perform a method for machine learning training with embedded visual labels, the method comprising:
retrieving, from the data store, a training data set, the training data set comprising a plurality of training data items and a plurality of classifications for the plurality of training data items;
creating a plurality of visual labels, each visual label of the plurality of visual labels representing a classification of the plurality of classifications for the plurality of training data items;
attaching each visual label of the plurality of visual labels to a training data item of the plurality of training data items to obtain a plurality of labeled training data items;
providing the plurality of labeled training data items to a machine learning model for training; and
for each labeled training data item of the plurality of labeled training data items:
receiving, from the machine learning model, a classification of the training data item; and
comparing the classification against the visual label.

### Claim 9 (depends on 8)

The system of claim 8, wherein the method further comprises separating, for each labeled training data item in the plurality of labeled training data items, the visual label from the training data item.

### Claim 10 (depends on 8)

The system of claim 8, wherein at least one visual label in the plurality of visual labels comprises a first classification of the training data item and a second classification of the training data item.

### Claim 11 (depends on 8)

The system of claim 8, wherein the method further comprises:
for each labeled training data item in the plurality of labeled training data items:
disposing the training data item on a first layer of the labeled training data item; and
disposing the visual label on a second layer of the labeled training data item.

### Claim 12 (depends on 8)

The system of claim 8, wherein the method further comprises in response to comparing the classification of the training data item to the visual label, updating the machine learning model.

### Claim 13 (depends on 8)

The system of claim 8, wherein the method further comprises storing metadata for a labeled training data item of the plurality of labeled training data items, the metadata comprising a coordinate location of the visual label in the labeled training data item.

### Claim 14 (depends on 8)

The system of claim 8, wherein each training data item in the plurality of training data items comprises a scanned image for training an optical character recognition machine learning model.

### Claim 15

A computer-implemented method for machine learning training with embedded visual labels, the computer-implemented method comprising:
generating a plurality of training data items for training a machine learning model;
in response to generating the plurality of training data items, creating a plurality of visual labels, each visual label of the plurality of visual labels identifying a training data item of the plurality of training data items;
attaching each visual label of the plurality of visual labels to a training data item of the plurality of training data items to obtain a plurality of labeled training data items;
for each labeled training data item of the plurality of labeled training data items, defining a separator to separate the visual label from the training data item;
providing the plurality of labeled training data items to the machine learning model for training; and
for each labeled training data item in the plurality of labeled training data items:
receiving, from the machine learning model, a classification of the training data item; and
comparing the classification against the visual label.

### Claim 16 (depends on 15)

The computer-implemented method of claim 15, wherein the visual label comprises a bar code.

### Claim 17 (depends on 15)

The computer-implemented method of claim 15,
wherein the training data item and the visual label comprise a greyscale image, and
wherein the separator comprises a colored separator.

### Claim 18 (depends on 15)

The computer-implemented method of claim 15, wherein each training data item of the plurality of training data items comprises an alphanumeric string.

### Claim 19 (depends on 18)

The computer-implemented method of claim 18, wherein the computer-implemented method further comprises:
for each labeled training data item in the plurality of labeled training data items:
altering the training data item within the labeled training data item to obtain an altered labeled training data item, the altering comprising at least one of a rotation, a smudging, a blurring, or an edit of the alphanumeric string of the alphanumeric string; and
adding the altered labeled training data item to the plurality of labeled training data items to form a training data set.

### Claim 20 (depends on 15)

The computer-implemented method of claim 15, wherein the computer-implemented method further comprises disposing each visual label of the plurality of visual labels in a predefined location of the labeled training data item.