Automatic benchmarking of labeling tasks

A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:

• selecting, from a set of labels for a data sample, a subset of the labels representing non-outliers in a distribution of values in the set of labels;
• aggregating the subset of the labels into a benchmark as a range of values that are valid for the data sample, wherein the aggregating includes,
• generating an outer boundary of the benchmark from a union of a set of polygons in the subset of the labels, and
• generating an inner boundary of the benchmark from an intersection of the set of polygons;
• generating, based on a comparison between the range of labels of the benchmark and an additional label, a benchmark score associated with the additional label for the data sample;
• and
• generating a set of performance metrics for labeling the data sample based on the benchmark score.

A system, comprising:

• a memory that stores instructions, and
• a processor that is coupled to the memory and, when executing the instructions, is configured to:
• select, from a set of labels for a data sample, a subset of the labels representing non-outliers in a distribution of values in the set of labels;
• aggregate the subset of the labels into a benchmark as a range of values that are valid for the data sample, wherein the aggregating includes,
• generating an outer boundary of the benchmark from a union of a set of polygons in the subset of the labels, and
• generating an inner boundary of the benchmark from an intersection of the set of polygons;
• generate, based on a comparison between the range of values of the benchmark and an additional label, a benchmark score associated with the additional label for the data sample;
• and
• generate a set of performance metrics for labeling the data sample based on the benchmark score.