US 12,762,205 B2Granted

Systems and methods for label quality assurance using consistency scores

AssigneeCapital One Services, LLC

InventorsGang Mei, Zhuqing Zhang

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods for improving label quality in datasets using a quality filter based on a consistency score. The system may receive a first given label for a first text string. It may process the first text string in an artificial intelligence model, as well as determine: a first predicted label for the first text string; and a first consistency score for the first text string based on a comparison of the first predicted label and the first given label. The system may compare the first consistency score to a first threshold consistency score and, in response to comparing the first consistency score to the first threshold consistency score, filter the first text string to a first group. The system may generate a recommendation to use the first group for a supervised learning task.

Claims20

Dependent claims

A system for improving label quality in datasets using a quality filter based on a consistency score, the system comprising:

cloud-based storage circuitry configured to store an artificial intelligence model trained to output predicted labels given inputted text strings;
cloud-based control circuitry configured to:
input a text string into the artificial intelligence model to determine a predicted label for the text string comprising a first probability distribution corresponding to a first set of candidate labels;
obtain a pre-existing label associated with the text string comprising a second probability distribution corresponding to a second set of candidate labels;
calculate a consistency score for the text string indicating a measure of agreement between the predicted label and the pre-existing label based on (i) a first comparison of a first measure of a center of the first probability distribution with a second measure of a center of the second probability distribution and (ii) a second comparison of a first variance metric of the first probability distribution with a second variance metric of the second probability distribution;
response to comparing the consistency score to a threshold consistency score, filter the text string to a group;
and
assign, to a training sample, the group as a label for a supervised learning task;
and
cloud-based input/output circuitry configured to:
display, on a user interface, the training sample for the supervised learning task.

8 more elements

A method, comprising:

inputting a first text string into an artificial intelligence model to predict a first label for the first text string, the first label comprising a first probability distribution corresponding to a first set of possible labels;
obtaining a second label associated with the first text string, the second label comprising a second probability distribution corresponding to a second set of possible labels;
calculating a first consistency score indicating a degree of consensus between the first label and the second label based on (i) a comparison of a first measure of a center of the first probability distribution with a second measure of a center of the second probability distribution and (i) a comparison of a first variance metric of the first probability distribution with a second variance metric of the second probability distribution;
comparing the first consistency score to the first a threshold consistency score;
filtering the first text string to a first group;
and
providing the first group as a label assigned to a training sample for a supervised learning task.

4 more elements

One or more non-transitory, computer readable medium comprising instructions that, when executed by one or more processors, causes operations comprising:

inputting a first text string into an artificial intelligence model, to obtain a first label for the first text string, the first label comprising a first probability distribution corresponding to a first set of labels;
obtaining a second label associated with the first text string, the second label comprising a second probability distribution corresponding to a second set of labels;
calculate a consistency score for the first text string based on a (i) a first comparison of a first measure of a center of the first probability distribution with a second measure of a center of the second probability distribution and (ii) a second comparison of a first variance metric of the first probability distribution with a second variance metric of the second probability distribution;
based on the consistency score, filtering the first text string to a first group;
and
providing the group as a label assigned to a training sample for a supervised learning task.

3 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/158,060Filed 23 Jan 2023
Published: US 2024/0248956 A125 Jul 2024
Granted: 13 Oct 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification4 codes

G06F 18/211
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: selection of the most…
G06F 18/2178
Validation; Performance evaluation; Active pattern learning techniques: based on feedback of a supervisor
G06F 40/30
Handling natural language data: semantic analysis
G06F
Computing and data processing