US 11,238,369 B2Granted

Interactive visualization evaluation for classification models

AssigneeInternational Business Machines Corporation

InventorsCagatay Demiralp, Marco Cavallo

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods and systems of classification model evaluation are described. A processor may generate a classifier track comprising visual indicators representing the set of classifier labels. The classifier labels may be based on output data of a classification model. The processor may generate a label track comprising visual indicators representing a set of observed labels received from a device. The processor may output the classifier track and the label track on a user interface. The processor may receive a request to evaluate the classification model, where the request may indicate a performance metric. The processor may identify a set of operators associated with the performance metric. The processor may execute the identified set of operators on the classifier track and the label track. The processor may generate a performance track indicating the performance metric of the classification model. The processor may output the performance track on the user interface.

Claims17

Dependent claims

A computer-implemented method comprising:

receiving, by a processor, output data from a classification model, wherein the output data comprises a set of probability scores of occurrences of an event within a range of time;
generating, by the processor, a set of classifier labels based on the output data, wherein the set of classifier labels is a set of discrete labels representing a portion of the output data;
receiving, by the processor, a set of observed labels from a device, wherein each observed label represents a ground-truth label of the occurrences of the event within the range of times;
generating, by the processor, a classifier track comprising a set of visual indicators representing the set of classifier labels, the set of classifier labels being arranged in the classifier track with respect to time;
generating, by the processor, a label track comprising a set of visual indicators representing the set of observed labels, the set of observed labels being arranged in the label track with respect to time;
outputting, by the processor, the classifier track and the label track on a user interface, wherein the classifier track and label track are displayed in a layout that aligns the classifier track and the label track with respect to time;
receiving, by the processor, a request for an evaluation of the classification model using the set of observed labels, wherein the request indicates a performance metric;
generating, by the processor, another label track comprising another set of visual indicators representing another set of observed labels;
executing, by the processor, an exclusive OR (XOR) operator on the label track and said another label track;
identifying, by the processor, at least one misclassified labels among the label track and said another label track based on the execution of the XOR operator;
and
removing, by the processor, the identified misclassified labels from the label track;
identifying, by the processor, a set of logical operators associated with the performance metric;
executing, by the processor, the identified set of logical operators on at least one of the classifier track and the label track with the removed misclassified labels;
generating, by the processor, a performance track indicating the performance metric of the classification model, wherein the performance track is generated based on a result of executing the identified set of logical operators on at least one of the classifier track and the label track with the removed misclassified labels;
and
outputting, by the processor, the performance track on the user interface to display the performance track, the classifier track, and label track in the layout, such that the performance track is aligned with the classifier track and the label track.

14 more elements

A system comprising:

a memory device configured to store:
output data from a classification model, wherein the output data comprises a set of probability scores of occurrences of an event within a range of time;
a set of observed labels, wherein each observed label represents a ground-truth label of the occurrences of the event within the range of times;
a hardware processor configured to be in communication with the memory device, the hardware processor being configured to:
obtain the output data from the memory device;
generate a set of classifier labels based on the output data, wherein the set of classifier labels is a set of discrete labels representing a portion of the output data;
obtain the set of observed labels from the memory device;
generate a classifier track comprising a set of visual indicators representing the set of classifier labels, the set of classifier labels being arranged in the classifier track with respect to time;
generate a label track comprising a set of visual indicators representing the set of observed labels, the set of observed labels being arranged in the label track with respect to time;
output the classifier track and the label track on a user interface, wherein the classifier track and label track are displayed in a layout that aligns the classifier track and the label track with respect to time;
receive a request for an evaluation of the classification model, wherein the request indicates a performance metric using the set of observed labels;
generate another label track comprising another set of visual indicators representing another set of observed labels;
execute an exclusive OR (XOR) operator on the label track and said another label track;
identify at least one misclassified labels among the label track and said another label track based on the execution of the XOR operator;
and
remove the identified misclassified labels from the label track;
identify a set of logical operators associated with the performance metric;
execute the identified set of logical operators on at least one of the classifier track and the label track with the removed misclassified labels;
generate a performance track indicating the performance metric of the classification model, wherein the performance track is generated based on a result of the execution of the identified set of logical operators;
and
output the performance track on the user interface to display the performance track, the classifier track, and label track in the layout, such that the performance track is aligned with the classifier track and the label track.

18 more elements

A computer program product of classification model evaluation, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processing element of a device to cause the device to:

receive output data from a classification model, wherein the output data comprises a set of probability scores of occurrences of an event within a range of time;
generate a set of classifier labels based on the output data, wherein the set of classifier labels is a set of discrete labels representing a portion of the output data;
receive a set of observed labels from a separate device, wherein each observed label represents a ground-truth label of the occurrences of the event within the range of times;
generate a classifier track comprising a set of visual indicators representing the set of classifier labels, the set of classifier labels being arranged in the classifier track with respect to time;
generate a label track comprising a set of visual indicators representing the set of observed labels, the set of observed labels being arranged in the label track with respect to time;
output the classifier track and the label track on a user interface, wherein the classifier track and label track are displayed in an layout that aligns the classifier track and the label track with respect to time;
receive a request for an evaluation of the classification model, wherein the request indicates a performance metric using the set of observed labels;
generate another label track comprising another set of visual indicators representing another set of observed labels;
execute an exclusive OR (XOR) operator on the label track and said another label track;
identify at least one misclassified labels among the label track and said another label track based on the execution of the XOR operator;
and
remove the identified misclassified labels from the label track;
identify a set of logical operators associated with the performance metric;
execute the identified set of logical operators on at least one of the classifier track and the label track with the removed misclassified labels;
generate a performance track indicating the performance metric of the classification model, wherein the performance track is generated based on a result of executing the identified set of logical operators;
and
output the performance track on the user interface to display the performance track, the classifier track, and label track in the layout, such that the performance track is aligned with the classifier track and the label track.

14 more elements

Dependencies are inferred from the claim text.