Method and system for unlabeled data selection using failed case analysis

An information handling system comprising:

• one or more processors;
• a memory coupled to at least one of the processors;
• a set of instructions stored in the memory and executed by at least one of the processors to automatically select training data for updating a model based on error type analysis, wherein the set of instructions are executable to perform actions of:
• applying, by the, human-annotated training data (Di) to a model to generate a plurality of results (Ri);
• evaluating, by the system, the plurality of results (Ri) to identify correct case results (C) and false case results (Fk);
• categorizing, by the system, the false case results (Fk) into one or more error type categories (Em) where the error type categories (Em) are selected from a group consisting of classifier misprediction, negation error, and incorrectly labeled ground truth;
• building, by the system, one or more error models (EM) corresponding to the one or more error type categories (Em), where each error model is built from at least failed case results belonging to a corresponding error type;
• applying, by the system, unlabeled data samples to the one or more error models (EM) to compute an error likelihood for each unlabeled data sample with respect to each error type category;
• selecting, by the system, one or more unlabeled data samples for annotation by a human subject matter expert based on a computed error likelihood for the one or more unlabeled data samples in a specified error type category meeting or exceeding an error threshold requirement;
• and
• displaying, by the system, the one or more unlabeled data samples as prioritized review candidates to solicit labeling feedback from the human subject matter expert.

A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by a processor at an information handling system, causes the system to automatically select training data for updating a model based on error type analysis by:

• applying human-annotated training data (Di) to a model to generate a plurality of results (Ri);
• evaluating the plurality of results (Ri) to identify correct case results (C) and false case results (Fk);
• categorizing the false case results (Fk) into one or more error type categories (Em) where the error type categories (Em) are selected from a group consisting of classifier misprediction, negation error, and incorrectly labeled ground truth;
• building one or more error models (EM) corresponding to the one or more error type categories (Em), where each error model is built from at least failed case results belonging to a corresponding error type;
• applying unlabeled data samples to the one or more error models (EM) to compute an error likelihood for each unlabeled data sample with respect to each error type category;
• selecting one or more unlabeled data samples for annotation by a human subject matter expert based on a computed error likelihood for the one or more unlabeled data samples in a specified error type category meeting or exceeding an error threshold requirement;
• and
• displaying the one or more unlabeled data samples as prioritized review candidates to solicit labeling feedback from the human subject matter expert.