Optimizing a machine learning system

A computer program product comprising a computer readable storage medium having program code embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, wherein the program code is readable and executable by a processor to perform a method of optimizing a machine learning system using semi-supervised learning, and wherein the method comprises:

• training a plurality of machine learning systems based on applying an initial labeled instance of data from an annotator;
• accessing a committee of classifiers, wherein the committee of classifiers is composed of the plurality of machine learning systems that have been trained using an initial labeled instance of data from an annotator, and wherein the initial labeled instance of data comprises attributes of the data, initial values of the attributes, and an initial prediction label that describes an initial predicted state based on the values;
• comparing an attributes ranking of the initial labeled instance of the data from the annotator to attributes rankings that are generated and used by each of the machine learning systems, wherein the comparing is performed upon evaluating one or more instances of unlabeled data that include the attributes, and wherein the attributes ranking of the initial labeled instance of the data comprises information indicative of a level of feature contributions for the instance of the data, and wherein the feature contributions describe an assessment by the annotator of how important a feature is to a prediction for a given instance of the data and to what direction, whether negative or positive contribution;
• weighting the machine learning systems according to how closely each of the attributes rankings generated by and used by each of the machine learning systems match the attributes ranking from the annotator, wherein the weighting defines weighted machine learning systems;
• inputting one or more subsequent instances of unlabeled data into the weighted machine learning systems to generate subsequent prediction labels for the one or more subsequent instances of unlabeled data, wherein the unlabeled data comprises the attributes of the data found in the initial labeled instance of data and new values of the attributes;
• further training the plurality of machine learning systems until an accuracy of the committee of classifiers reaches a defined threshold, wherein the further training comprises applying the subsequent prediction labels, and wherein reaching the defined threshold defines the committee of classifiers as being trained to accurately generate prediction labels for future unlabeled incidents of data that have the attributes of the initial incident of data;
• assigning prediction labels to incoming unlabeled instances of data using the committee of classifiers that has reached the defined threshold;
• and
• calculating, by the computing device, a level of disparity among the committee of classifiers when evaluating the one or more subsequent instances of unlabeled data, wherein the disparity is based on both differences in the subsequent prediction labels and differences in confidence in the subsequent prediction labels.

A computer system comprising one or more processors, one or more computer readable memories, and one or more computer readable non-transitory storage mediums, and program instructions stored on at least one of the one or more computer readable non-transitory storage mediums for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the stored program instructions executed to perform a method comprising:

• training a plurality of machine learning systems based on applying an initial labeled instance of data from an annotator;
• accessing a committee of classifiers, wherein the committee of classifiers is composed of the plurality of machine learning systems that have been trained using an initial labeled instance of data from an annotator, and wherein the initial labeled instance of data comprises attributes of the data, initial values of the attributes, and an initial prediction label that describes an initial predicted state based on the values;
• receiving an attributes ranking of the initial labeled instance of the data from the annotator, wherein the attributes ranking is based on a determination by the annotator of an impact level that each attribute has when generating a prediction label for an instance of data that has the attributes of the initial labeled instance of data;
• comparing the attributes ranking from the annotator to attributes rankings that are generated and used by each of the machine learning systems, wherein the comparing is performed upon evaluating one or more instances of unlabeled data that include the attributes;
• weighting the machine learning systems according to how closely each of the attributes rankings generated by and used by each of the machine learning systems match the attributes ranking from the annotator, wherein the weighting defines weighted machine learning systems;
• inputting one or more subsequent instances of unlabeled data into the weighted machine learning systems to generate subsequent prediction labels for the one or more subsequent instances of unlabeled data, wherein the unlabeled data comprises the attributes of the data found in the initial labeled instance of data and new values of the attributes;
• further training the plurality of machine learning systems until an accuracy of the committee of classifiers reaches a defined threshold, wherein the further training comprises applying the subsequent prediction labels, and wherein reaching the defined threshold defines the committee of classifiers as being trained to accurately generate prediction labels for future unlabeled incidents of data that have the attributes of the initial incident of data;
• assigning prediction labels to incoming unlabeled instances of data using the committee of classifiers that has reached the defined threshold;
• and
• calculating, by the computing device, a level of disparity among the committee of classifiers when evaluating the one or more subsequent instances of unlabeled data, wherein the disparity is based on differences in the subsequent prediction labels and differences in confidence in the subsequent prediction labels.