# US 2024/0005099 A1 · Integrated synthetic labeling optimization for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005099 A1
- Document ID: US20240005099A1
- Status: Published application
- Kind: A1
- Application number: 17/810,123
- Filed: 2022-06-30
- Published: 2024-01-04
- Earliest claimed: 2022-06-30
- Assignees: Paypal, Inc.
- Applicants: PayPal, Inc.
- Inventors: Alon Dourban; Roy Lothan; Myriam Lesmy; Maya Cohen; Itay Margolin
- Web page: https://patentgrep.com/patent/US20240005099A1

## CPC

- G06F 40/30: Semantic analysis
- G06N 20/00: Machine learning
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/279: Recognition of textual entities
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are disclosed relating to weakly supervised machine learning, which may be employed when there is a limited amount of labeled data available. A computer system may generate respective sets of synthetic labels for unlabeled data for a classification problem, where a given set of synthetic labels is produced by a corresponding one of a plurality of different label models. The computer system may then fit a set of supervised models, where each supervised model is fitted with one of the respective sets of synthetic labels to produce a respective set of predictions. The computer system may then evaluate the set of supervised models based on their respective set of predictions and using a set of labeled data for the classification problem. The evaluation may be used to select a particular supervised model and its corresponding label model.

## Claims

### Claim 1

A method, comprising:
generating, by a computer system, respective sets of synthetic labels for unlabeled data for a classification problem, wherein a given one of the respective sets of synthetic labels is produced by a corresponding one of a plurality of different label models;
fitting, by the computer system, a set of supervised models, a given one of the set of supervised models being fitted with one of the respective sets of synthetic labels to produce a respective set of predictions; and
evaluating, by the computer system, the set of supervised models based on their respective set of predictions and using a set of labeled data for the classification problem.

### Claim 2 (depends on 1)

The method of claim 1, wherein the classification problem is a text-classification problem.

### Claim 3 (depends on 2)

The method of claim 2, wherein each supervised model is also fitted using a set of general features.

### Claim 4 (depends on 3)

The method of claim 3, wherein the set of general features specifies values for a word embedding vector.

### Claim 5 (depends on 2)

The method of claim 2, wherein the text-classification problem is sentiment analysis.

### Claim 6 (depends on 1)

The method of claim 1, wherein the different label models utilize different subsets of a set of rules.

### Claim 7 (depends on 1)

The method of claim 1, wherein the evaluating includes evaluating the respective set of predictions according to the set of labeled data.

### Claim 8 (depends on 1)

The method of claim 1, further comprising selecting, by the computer system, at least one of the set of supervised models based on the evaluating.

### Claim 9 (depends on 1)

The method of claim 1, wherein the classification problem is a text-classification problem, and wherein a size of the set of labeled data is sufficient to evaluate, but not train, the set of supervised models.

### Claim 10 (depends on 9)

The method of claim 9, wherein the different label models utilize different subsets of a set of rules.

### Claim 11 (depends on 10)

The method of claim 10, wherein the evaluating includes evaluating the respective set of predictions according to the set of labeled data.

### Claim 12

A non-transitory computer-readable medium having instructions stored thereon that are executable by a computer system to perform operations for weakly supervised machine learning, the operations comprising:
jointly optimizing a desired label model and a corresponding desired supervised model for a classification problem, wherein the jointly optimizing includes:
for each of a plurality of label models having different rule sets, generating, from a set of unlabeled data for the classification problem, a respective set of synthetic labels;
fitting each of a plurality of supervised models with one of the respective sets of synthetic labels to generate respective sets of predictions; and
evaluating, using a set of labeled data, the plurality of supervised models using the respective sets of predictions; and
wherein the evaluating is usable to select a particular label model and the corresponding particular supervised model from the plurality of label models and the plurality of supervised models.

### Claim 13 (depends on 12)

The computer-readable medium of claim 12, wherein the different rule sets are different subsets of a plurality of domain-specific heuristics for the classification problem.

### Claim 14 (depends on 12)

The computer-readable medium of claim 12, wherein the classification problem is a text-classification problem, and wherein each of the plurality of supervised models is also fitted using a set of vector values for terms in a word embedding space.

### Claim 15 (depends on 12)

The computer-readable medium of claim 12, wherein the classification problem is a text-classification problem, wherein a size of the set of labeled data is sufficient to evaluate the respective sets of predictions of the plurality of supervised models, but is insufficient to train the plurality of supervised models.

### Claim 16 (depends on 12)

The computer-readable medium of claim 12, wherein the operations further comprise:
utilizing the particular supervised model to evaluate the classification problem for subsequently generated data samples.

### Claim 17

A method, comprising:
receiving, by a computer system, a data sample for a classification problem;
accessing, by the computer system, a particular supervised model and a corresponding label model that were selected, prior to the accessing, by:
generating respective sets of intermediate labels for unlabeled data for the classification problem, wherein a given set of intermediate labels is produced by a corresponding one of a plurality of different label models;
fitting a set of supervised models, each supervised model being fitted with one of the respective sets of intermediate labels to produce a respective set of predictions; and
evaluating, using a set of labeled data for the classification problem, the set of supervised models based on their respective set of predictions; and
selecting, based on the evaluating, the particular supervised model and the corresponding label model; and
classifying, by the computer system, the received data sample using the particular supervised model and the corresponding label model.

### Claim 18 (depends on 17)

The method of claim 17, wherein the classification problem is a text-classification problem, and wherein each of the set of supervised models are also fitted using a set of general features.

### Claim 19 (depends on 18)

The method of claim 18, wherein the set of general features specifies values for a word embedding vector.

### Claim 20 (depends on 17)

The method of claim 17, wherein the classification problem is a text-classification problem, and wherein a size of the set of labeled data is sufficient to evaluate, but not train, the set of supervised models.
