# US 2021/0334468 A1 · Feature Reweighting in Text Classifier Generation Using Unlabeled Data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334468 A1
- Document ID: US20210334468A1
- Status: Published application
- Kind: A1
- Application number: 16/860,565
- Filed: 2020-04-28
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Yang Yu; Haode Qi; Haoyu Wang; Ming Tan; Navneet N. Rao; Saloni Potdar; Robert Leslie Yates
- Web page: https://patentgrep.com/patent/US20210334468A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 16/35: Clustering; Classification
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A mechanism is provided to implement a text classifier training augmentation mechanism for incorporating unlabeled data into the generation of a text classifier. For each term of a plurality of terms in each document of a plurality of documents in a set of unlabeled data, a term frequency value is determined. The term is normalized by dividing the term frequency value by a total number of terms in the document. An inverse document frequency (idf) value is determined for each term based on the term frequency value. A subset of terms is filtered from the plurality of terms based the determined idf values. The idf values for the remaining terms are transformed into feature weights. Terms from a set of labeled data are re-weighted based on the feature weights determined from the set of unlabeled data. The text classifier is then generated using the re-weighted labeled data.

## Claims

### Claim 1

A method, in a data processing system, comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that are executed by the at least one processor to configure the at least one processor to implement a text classifier training augmentation mechanism for incorporating unlabeled data in addition to labeled data into the generation of a text classifier, the method comprising:
for each term of a plurality of terms in each document of a plurality of documents in a set of unlabeled data, determining, by the text classifier training augmentation mechanism, a term frequency value;
normalizing, by the text classifier training augmentation mechanism, the term by dividing the term frequency value by a total number of terms in the document;
determining, by the text classifier training augmentation mechanism, an inverse document frequency (idf) value for each term in the plurality of terms based on the term frequency value;
filtering, by the text classifier training augmentation mechanism, a subset of terms from the plurality of terms based the determined idf values;
transforming, by the text classifier training augmentation mechanism, the idf values for the remaining terms into feature weights;
re-weighting, by the text classifier training augmentation mechanism, terms from a set of labeled data based on the feature weights determined from the set of unlabeled data; and
generating, by a machine learning mechanism, the text classifier using the re-weighted labeled data.

### Claim 2 (depends on 2)

The method of claim 2, further comprising:
weighing down, by the text classifier training augmentation mechanism, frequent terms in the plurality of terms while scaling up rare terms in the plurality of terms by computing the idf value for the term using the following equation:
IDF
⁡
(
t
)
=
log
⁢
⁢
_
⁢
⁢
e
(
Total
⁢
⁢
⁢
number
⁢
⁢
of
⁢
⁢
⁢
documents
Number
⁢
⁢
of
⁢
⁢
documents
⁢
⁢
with
⁢
⁢
term
⁢
⁢
t
)
.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating, by the text classifier training augmentation mechanism, a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data;
computing, by the text classifier training augmentation mechanism, a cosine similarity between each sentence representation of a plurality of sentences in the set of labeled data and each sentence representation of the set of unlabeled data;
determining, by the text classifier training augmentation mechanism, a weighted sum of the similarities for each sentence in the set of labeled data;
normalizing, by the text classifier training augmentation mechanism, the weighted sums over all the plurality of sentences in the labeled data thereby producing a set of normalized sample reweights; and
generating, by the machine learning mechanism, the text classifier using the set of normalized sample reweights.

### Claim 4 (depends on 3)

The method of claim 3, wherein the machine learning mechanism applies each reweight as an additional training objective in generating the text classifier such that:
Loss(original example)+alpha*Loss(weighted example)
where alpha is a hyperparameter that is user configurable.

### Claim 5 (depends on 4)

The method of claim 4, wherein the Loss(weighted example) is equal to a learned-weight multiplied by the Loss(original example).

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating, by the text classifier training augmentation mechanism, a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data thereby generating a set of augmented sentences;
identifying, by the text classifier training augmentation mechanism, a prediction distribution between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data;
performing, by the text classifier training augmentation mechanism, an inter-sample agreement check to identify a consistency loss value between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data; and
generating, by the machine learning mechanism, the text classifier using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value.

### Claim 7 (depends on 6)

The method of claim 6, wherein the machine learning mechanism generates the text classifier using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value using the following equation:
Loss(original example)+alpha*Loss(weighted example)+gamma*Consistency_loss(unlabeled samples)
where alpha and gamma are hyperparameters that are user configurable.

### Claim 8 (depends on 7)

The method of claim 7, wherein the Loss(weighted example) is equal to a learned-weight multiplied by the Loss(original example) and wherein the Consistency_loss is an inverse to a similarity between at least one generated sentence and at least one associated unlabeled sentence.

### Claim 9

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a text classifier training augmentation mechanism for incorporating unlabeled data in addition to labeled data into the generation of a text classifier, and further causes the data processing system to:
for each term of a plurality of terms in each document of a plurality of documents in a set of unlabeled data, determine a term frequency value;
normalize the term by dividing the term frequency value by a total number of terms in the document;
determine an inverse document frequency (idf) value for each term in the plurality of terms based on the term frequency value;
filter a subset of terms from the plurality of terms based the determined idf values;
transform the idf values for the remaining terms into feature weights;
re-weight terms from a set of labeled data based on the feature weights determined from the set of unlabeled data; and
generate the text classifier using the re-weighted labeled data.

### Claim 10 (depends on 9)

The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:
weigh down frequent terms in the plurality of terms while scaling up rare terms in the plurality of terms by computing the idf value for the term using the following equation:
IDF
⁡
(
t
)
=
log
⁢
⁢
_
⁢
⁢
e
(
Total
⁢
⁢
⁢
number
⁢
⁢
of
⁢
⁢
⁢
documents
Number
⁢
⁢
of
⁢
⁢
documents
⁢
⁢
with
⁢
⁢
term
⁢
⁢
t
)
.

### Claim 11 (depends on 9)

The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:
generate a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data;
compute a cosine similarity between each sentence representation of a plurality of sentences in the set of labeled data and each sentence representation of the set of unlabeled data;
determine a weighted sum of the similarities for each sentence in the set of labeled data;
normalize the weighted sums over all the plurality of sentences in the labeled data thereby producing a set of normalized sample reweights; and
generate the text classifier using the set of normalized sample reweights.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein each reweight is applied as an additional training objective in generating the text classifier such that:
Loss(original example)+alpha*Loss(weighted example)
where alpha is a hyperparameter that is user configurable.

### Claim 13 (depends on 9)

The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:
generate a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data thereby generating a set of augmented sentences;
identify a prediction distribution between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data;
perform an inter-sample agreement check to identify a consistency loss value between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data; and
generate the text classifier using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein the text classifier is generated using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value using the following equation:
Loss(original example)+alpha*Loss(weighted example)+gamma*Consistency_loss(unlabeled samples)
where alpha and gamma are hyperparameters that are user configurable.

### Claim 15

An apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement a text classifier training augmentation mechanism for incorporating unlabeled data in addition to labeled data into the generation of a text classifier, and further cause the at least one processor to:
for each term of a plurality of terms in each document of a plurality of documents in a set of unlabeled data, determine a term frequency value;
normalize the term by dividing the term frequency value by a total number of terms in the document;
determine an inverse document frequency (idf) value for each term in the plurality of terms based on the term frequency value;
filter a subset of terms from the plurality of terms based the determined idf values;
transform the idf values for the remaining terms into feature weights;
re-weight terms from a set of labeled data based on the feature weights determined from the set of unlabeled data; and
generate the text classifier using the re-weighted labeled data.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the instructions further cause the at least one processor to:
weigh down frequent terms in the plurality of terms while scaling up rare terms in the plurality of terms by computing the idf value for the term using the following equation:
IDF
⁡
(
t
)
=
log
⁢
⁢
_
⁢
⁢
e
(
Total
⁢
⁢
⁢
number
⁢
⁢
of
⁢
⁢
⁢
documents
Number
⁢
⁢
of
⁢
⁢
documents
⁢
⁢
with
⁢
⁢
term
⁢
⁢
t
)
.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the instructions further cause the at least one processor to:
generate a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data;
compute a cosine similarity between each sentence representation of a plurality of sentences in the set of labeled data and each sentence representation of the set of unlabeled data;
determine a weighted sum of the similarities for each sentence in the set of labeled data;
normalize the weighted sums over all the plurality of sentences in the labeled data thereby producing a set of normalized sample reweights; and
generate the text classifier using the set of normalized sample reweights.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein each reweight is applied as an additional training objective in generating the text classifier such that:
Loss(original example)+alpha*Loss(weighted example)
where alpha is a hyperparameter that is user configurable.

### Claim 19 (depends on 15)

The apparatus of claim 15, wherein the instructions further cause the at least one processor to:
generate a sentence representation for each sentence of a plurality of sentences in each document of the plurality of documents in the set of unlabeled data thereby generating a set of augmented sentences;
identify a prediction distribution between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data;
perform an inter-sample agreement check to identify a consistency loss value between the plurality of sentences in the set of unlabeled data and the set of augmented sentences from the set of unlabeled data; and
generate the text classifier using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the text classifier is generated using the plurality of sentences in the set of unlabeled data, the set of augmented sentences from the set of unlabeled data, and the consistency loss value using the following equation:
Loss(original example)+alpha*Loss(weighted example)+gamma*Consistency_loss(unlabeled samples)
where alpha and gamma are hyperparameters that are user configurable.
