Learning with noisy labels from multiple judges

A system for determining a plurality of inferred labels, for a corresponding plurality of items, wherein the inferred labels are generated from a plurality of judgments, wherein multiple judges select the judgments from a specified choice of labels for each of the items, comprising:

• a processing unit;
• and
• a system memory, wherein the system memory comprises code configured to direct the processing unit to:
• determine a true label for each of the items based on the judgments;
• determine a characterization of labeling expertise for each judge based on the judgments;
• determine a labeling difficulty for each item based on the judgments;
• determine, using maximum entropy, a probability distribution over the specified choice of labels for each judge and item;
• selecting a plurality of improved labels for the items from the specified choice of labels such that the entropy over the probability distribution is reduced, wherein the improved labels represent an improvement from the judgments toward the true labels;
• and
• perform iterative procedure wherein the true labels, characterizations of judge expertise, and the labeling difficulties are determined.

One or more computer-readable storage media for determining a plurality of inferred labels, for a corresponding plurality of items, wherein the inferred labels are generated from a plurality of judgments, wherein multiple judges select the judgments from a specified choice of labels for each of the items, the computer-readable storage media comprising code configured to direct a processing unit to:

• determine a true label for each of the items based on the judgments;
• determine a characterization of labeling expertise for each of the judges based on the judgments, wherein the expertise of each judge represents a relative expertise in relation to all other judges, and wherein the expertise is independent of any item a judge labels;
• determine a labeling difficulty for each of the items based on the judgments, wherein the labeling difficulty for each of the items represents a relative difficulty in comparison to all other items, and wherein the labeling difficulty for each of the items is independent of any judge labeling an item;
• determine, using maximum entropy, a probability distribution over the specified choice of labels for each judge and item, based on the judgments;
• select a plurality of improved labels for the items from the specified choice of labels such that the entropy over the probability distribution is reduced, wherein the improved labels represent an improvement from the judgments toward the true labels;
• and
• perform an iterative procedure wherein the true labels, the characterizations of labeling expertise, and the labeling difficulties are determined.