# US 2021/0334596 A1 · Automatically labeling data using conceptual descriptions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334596 A1
- Document ID: US20210334596A1
- Status: Published application
- Kind: A1
- Application number: 16/859,695
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Haibo Ding; Zhe Feng
- Web page: https://patentgrep.com/patent/US20210334596A1

## CPC

- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system for automatically labeling data using conceptual descriptions. In one example, the system includes an electronic processor configured to generate unlabeled training data examples from one or more natural language documents and, for each of a plurality of categories, determine one or more concepts associated with a conceptual description of the category and generate a weak annotator for each of the one or more concepts. The electronic processor is also configured to apply each weak annotator to each training data example and, when a training data example satisfies a weak annotator, output a category associated with the weak annotator. For each training data example, the electronic processor determines a probabilistic distribution of the plurality of categories. For each training data example, the electronic processor labels the training data example with a category having the highest value in the probabilistic distribution determined for the training data example.

## Claims

### Claim 1

A system for automatically labeling data using conceptual descriptions, the system comprising an electronic processor configured to
generate unlabeled training data examples from one or more natural language documents;
for each of a plurality of categories,
determine one or more concepts associated with a conceptual description of the category; and
generate a weak annotator for each of the one or more concepts; and
apply each weak annotator to each training data example and, when a training data example satisfies a weak annotator, output a category associated with the weak annotator;
for each training data example, determine a probabilistic distribution of the plurality of categories, the probabilistic distribution representing, for each of the plurality of categories, a likelihood that the category is a correct label for the training data example; and
for each training data example, label the training data example with a category having the highest value in the probabilistic distribution determined for the training data example.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the electronic processor is further configured to select training data examples, based on the probabilistic distributions associated with the training data examples, to use to train a machine learning system.

### Claim 3 (depends on 2)

The system according to claim 2, wherein the electronic processor is further configured to select training data examples to use to train a machine learning system when a value included in a probabilistic distribution determined for a training data example is above a predetermined threshold.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the electronic processor is further configured to produce one or more natural language documents in a state in which training data examples may be extracted from the one or more natural language documents by
extracting text from the natural language documents;
tokenizing the extracted text;
splitting the text into sentences;
annotating each token with a part of speech tag; and
annotating the dependency relations for pairs of words.

### Claim 5 (depends on 1)

The system according to claim 1, wherein the electronic processor is configured to apply each weak annotator to each training data example and, when a training data example satisfies a weak annotator, output a category associated with the weak annotator by
outputting a category associated with a concept for a training data example, when the training data example contains a term that is an instance of the concept.

### Claim 6 (depends on 1)

The system according to claim 1, wherein the electronic processor is configured to apply each weak annotator to each training data example and, when a training data example satisfies a weak annotator, output a category associated with the weak annotator by
computing, using word embeddings, a similarity between a concept associated with a weak annotator and a training data example; and
based on the computed similarity, determining whether to output a category for a training data example.

### Claim 7 (depends on 1)

The system according to claim 1, wherein a training data example is a tuple including parts of speech of a natural language sentence.

### Claim 8 (depends on 1)

The system according to claim 1, wherein the electronic processor is further configured to train the machine learning system using the labeled training examples; and
execute the trained machine learning system to categorize a natural language passage.

### Claim 9 (depends on 1)

The system according to claim 1, wherein the electronic processor is further configured to remove noisy concepts from the one or more concepts.

### Claim 10

A method for automatically labeling data using conceptual descriptions, the method comprising:
generating, with an electronic processor, unlabeled training data examples from one or more natural language documents;
for each of a plurality of categories,
determining one or more concepts associated with a conceptual description of the category; and
generating a weak annotator for each of the one or more concepts; and
applying each weak annotator to each training data example and, when a training data example satisfies a weak annotator, outputting a category associated with the weak annotator;
for each training data example, determining a probabilistic distribution of the plurality of categories, the probabilistic distribution representing, for each of the plurality of categories, a likelihood that the category is a correct label for the training data example; and
for each training data example, labeling the training data example with a category having the highest value in the probabilistic distribution determined for the training data example.

### Claim 11 (depends on 10)

The method according to claim 10, the method further comprising selecting training data examples, based on the probabilistic distributions associated with the training data examples, to use to train a machine learning system.

### Claim 12 (depends on 11)

The method according to claim 11, the method further comprising selecting training data examples to use to train a machine learning system when a value included in a probabilistic distribution determined for a training data example is above a predetermined threshold.

### Claim 13 (depends on 10)

The method according to claim 10, the method further comprising producing one or more natural language documents in a state in which training data examples may be extracted from the one or more natural language documents by
extracting text from the natural language documents;
tokenizing the extracted text;
splitting the text into sentences;
annotating each token with a part of speech tag; and
annotating the dependency relations for pairs of words.

### Claim 14 (depends on 10)

The method according to claim 10, wherein applying each weak annotator to each training data example and, when a training data example satisfies a weak annotator, outputting a category associated with the weak annotator includes
outputting a category associated with a concept for a training data example, when the training data example contains a term that is an instance of the concept.

### Claim 15 (depends on 10)

The method according to claim 10, wherein applying each weak annotator to each training data example and, when a training data example satisfies a weak annotator, outputting a category associated with the weak annotator includes
computing, using word embeddings, a similarity between a concept associated with a weak annotator and a training data example; and
based on the computed similarity, determining whether to output a category for a training data example.

### Claim 16 (depends on 10)

The method according to claim 10, wherein a training data example is a tuple including parts of speech of a natural language sentence.

### Claim 17 (depends on 10)

The method according to claim 10, the method further comprising:
training the machine learning system using the labeled training examples; and
executing the trained machine learning system to categorize a natural language passage.

### Claim 18 (depends on 10)

The method according to claim 10, the method further comprising removing noisy concepts from the one or more concepts.
