# US 12,182,733 B2 · Label inference system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,733 B2
- Document ID: US12182733B2
- Status: Granted
- Kind: B2
- Application number: 17/394,641
- Filed: 2021-08-05
- Published: 2022-03-17
- Granted: 2024-12-31
- Earliest claimed: 2020-09-15
- Assignees: Vinbrain Joint Stock Company
- Applicants: Vinbrain Joint Stock Company
- Inventors: Chanh DT. Nguyen; Hoang N. Nguyen; Thanh M. Huynh; Steven QH. Truong
- Web page: https://patentgrep.com/patent/US12182733B2

## CPC

- G06N 5/04: Inference or reasoning models
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 20/00: Machine learning
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Provided is a label inference system including a data generator configured to generate a training set and a test set, each including a plurality of images labeled with experts' annotations, a data trainer configured to perform training for a base model based on the generated training set and test set, a determiner configured to identify whether an evaluation metric f1 of the training model satisfies a base evaluation metric f1base, and a data inference unit configured to perform inference using the training set, the test set, and an unlabeled data set with the training model satisfying the base evaluation metric f1base.

## Claims

### Claim 1

A label inference system comprising:
a data generator configured to generate a training set and a test set, each including a plurality of images labeled with experts' annotations;
a data trainer configured to perform training for a base model based on the generated training set and test set;
a determiner configured to identify whether an evaluation metric f1 of the training model satisfies a base evaluation metric f1base; and
a data inference unit configured to perform inference using the training set, the test set, and an unlabeled data set with the training model satisfying the base evaluation metric f1base,
wherein the data inference unit calculates a current inference value of the unlabeled data set from a weighted average value of a previous inference value and a current estimated value according to the following Equation 1:
{circumflex over (p)}t=Fμ({circumflex over (p)}t-1,Pt), 1
wherein {circumflex over (p)}t represents the current inference value, {circumflex over (p)}t-1 represents a previous inference value, pt represents a current estimated value, Fμrepresents an online update operator with momentum μ, and t is a natural number.

### Claim 2 (depends on 1)

The label inference system of claim 1, wherein the data generator expands the training set by labeling and adding another batch of data into the training set when the determiner identifies that the evaluation metric f1 of the training model is less than or equal to the base evaluation metric f1base.

### Claim 3 (depends on 2)

The label inference system of claim 2, wherein the data trainer performs for the base model based on expanded training set and the test set.

### Claim 4 (depends on 1)

The label inference system of claim 1, wherein, when the determiner identifies that the evaluation metric f1 of the training model is greater than the base evaluation metric f1base, the data inference unit selects an unlabeled data set from a data pool.

### Claim 5 (depends on 1)

The label inference system of claim 1, wherein the current inference values {circumflex over (p)}t comprise samples with high confident score {circumflex over (p)}tH and samples with low confident score {circumflex over (p)}tL.

### Claim 6 (depends on 5)

The label inference system of claim 5, wherein the data inference unit extracts samples with thresholded high confidence scores {circumflex over (p)}tH,thresh by hard-thresholding a subset of the samples with the high confidence scores {circumflex over (p)}tH using a threshold τt.

### Claim 7 (depends on 6)

The label inference system of claim 6, wherein the data inference unit extracts samples with relabeled low confidence score {circumflex over (p)}tL,relabel, which is a subset of the samples with low confidence scores relabeled by an user.

### Claim 8 (depends on 7)

The label inference system of claim 7, wherein the data inference unit extracts outliers of the training set and the test set, which need to be relabeled by a user, using the threshold τt.

### Claim 9 (depends on 8)

The label inference system of claim 8, wherein the data inference unit performs inference using the updated training set and test set, which are modified by relabeling, and the updated unlabeled data set.

### Claim 10 (depends on 9)

The label inference system of claim 9, wherein the data inference unit trains a new snapshot of the training model for some iterations t using the updated unlabeled data set, training set, and test set.

### Claim 11 (depends on 10)

The label inference system of claim 10, wherein, when the determiner identifies that the evaluation metric f1 of the training model is greater than the target evaluation metric f1target, a terminal condition of the data inference unit is triggered.

### Claim 12 (depends on 1)

The label inference system of claim 1, wherein the data inference unit calculates the current inference value of the unlabeled data set using momentum.

### Claim 13 (depends on 1)

The label inference system of claim 1, wherein the momentum is set to 0.5.