# US 2021/0241153 A1 · Method and system for improving quality of a dataset

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241153 A1
- Document ID: US20210241153A1
- Status: Published application
- Kind: A1
- Application number: 16/779,522
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Applicants: Element Al Inc.
- Inventors: Frédéric BRANCHAUD-CHARRON; Parmida ATIGHEHCHIAN; Jan FREYBERG; Lorne SCHELL
- Web page: https://patentgrep.com/patent/US20210241153A1

## CPC

- G06N 20/00: Machine learning
- G06F 16/285: {Clustering or classification}
- G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/25: Fusion techniques
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method and a server for updating a dynamic list of labeling tasks. One or more labels are received, each label associated to one labeling task; the one or more received labels are inserted into a dataset; an artificial intelligence (AI) model is trained on labeled data items from the dataset; predicted labels are obtained for a plurality of unlabeled data items from the dataset by applying the model thereon; a model-uncertainty measurement is computed by applying one or more regularization methods; relevancy values are computed for at least a subset of the predicted labels taking into account the predicted label and the model-uncertainty measurement; the data items corresponding to the labeling tasks with the highest relevancy values are inserted in the dynamic list; and the dynamic list is reordered upon computing of the relevancy values.

## Claims

### Claim 1

A method for updating a dynamic list of labeling tasks, the dynamic list comprising data items for which labeling tasks are to be performed and a relevancy value for each of the labeling tasks therein, the method comprising:
at a server, receiving one or more labels, each label associated to one labeling task;
at the server, inserting the one or more received labels into a dataset;
at the server, training an Artificial Intelligence (AI) model on labeled data items from the dataset;
obtaining predicted labels for a plurality of unlabeled data items from the dataset by applying the model thereon;
computing a model-uncertainty measurement by applying one or more uncertainty estimation methods;
computing relevancy values for at least a subset of the predicted labels taking into account the predicted label and the model-uncertainty measurement;
inserting in the dynamic list, the data items corresponding to the labeling tasks with the highest relevancy values; and
reordering the dynamic list upon computing of the relevancy values.

### Claim 2 (depends on 1)

The method of claim 1, wherein examples of uncertainty estimation methods comprise Monte-Carlo Dropout and Bayesian Network.

### Claim 3 (depends on 1)

The method of claim 1 further comprising assigning tasks from the dynamic list to labelers considering relevancy value of the predicted labels.

### Claim 4 (depends on 1)

The method of claim 1 further comprising re-computing the dynamic list based on triggers comprising one or more of: a number of idle processing cycles and a magnitude of the highest relevancy values.

### Claim 5

An artificial intelligence server configured for updating a dynamic list of labeling tasks to be performed on data items of a dataset stored in a storage system, the artificial intelligence server comprising:
a memory module for storing versions of the dynamic list;
a processor module configured to:
receive one or more labels, each label associated to one labeling task;
insert the one or more received labels into a dataset;
train an Artificial Intelligence (AI) model on labeled data items from the dataset;
obtain predicted labels for a plurality of unlabeled data items from the dataset by applying the model thereon;
compute a model-uncertainty measurement by applying one or more uncertainty estimation methods;
compute relevancy values for at least a subset of the predicted labels taking into account the predicted label and the model-uncertainty measurement;
insert in the dynamic list, the data items corresponding to the labeling tasks with the highest relevancy values; and
reorder the dynamic list upon computing of the relevancy values.

### Claim 6 (depends on 5)

The artificial intelligence server of claim 5, wherein the processor module is further for assigning tasks from the dynamic list to labelers considering relevancy value of the predicted labels.

### Claim 7 (depends on 5)

The artificial intelligence server of claim 5, wherein the processor module is further for re-computing the dynamic list based on triggers comprising one or more of: a number of idle processing cycles and a magnitude of the highest relevancy values.

### Claim 8 (depends on 5)

The artificial intelligence server of claim 5, further comprising a network interface module for interfacing with a plurality of remote labelers.

### Claim 9 (depends on 5)

The artificial intelligence server of claim 5, further comprising a network interface module for communicating the dynamic list to labelers.

### Claim 10 (depends on 5)

The artificial intelligence server of claim 5, further comprising a network interface module for communicating the labels to the processor module.

### Claim 11 (depends on 5)

The artificial intelligence server of claim 5, wherein the storage module is a remote database accessible through a network.