Machine Learning Classification with Confidence Thresholds

A method comprising:

• obtaining, by a computing system, a plurality of confidence thresholds for predictions made by a machine learning classifier, wherein the machine learning classifier is configured to:
• receive input observations;
• predict, for each respective input observation, a set of probabilities that indicate a likelihood of each respective input observation belonging to each respective output category of a plurality of output categories;
• and
• classify each respective input observation into the output category having the highest probability of the plurality of output categories;
• reclassifying, for each confidence threshold of a plurality of confidence thresholds, by the computing system, any of the input observations for which all probabilities of the set of probabilities are less than the corresponding confidence threshold into a null category that is not one of the output categories;
• after the reclassifying, determining, by the computing system, for each confidence threshold of the plurality of confidence thresholds and for each of the plurality of output categories, a precision value that indicates a percentage of input observations correctly predicted to be in a particular output category out of a total number of input observations predicted to be in the particular output category;
• after the reclassifying, determining, by the computing system, for each confidence threshold of the plurality of confidence thresholds and for each of the plurality of output categories, a coverage value that indicates a percentage of non-null-category predictions made for the input observations in the particular output category;
• after the reclassifying, determining, by the computing system, for each confidence threshold of the plurality of confidence thresholds and for each of the plurality of output categories, an objective function value, wherein the objective function value is a multiplicative product of the precision value and the coverage value;
• generating a graphical user interface for display, wherein the graphical user interface visually associates confidence thresholds, precision values, coverage values, and objective function values for each of the plurality of output categories, wherein the confidence threshold is one of a plurality of confidence thresholds on the graphical user interface, and wherein each of the plurality of confidence thresholds are selectably displayed with the associated precision values, the associated coverage values, and the associated objective function values;
• and
• performing, by at least the machine learning classifier, further classification of additional input observations into the output categories and the null category in accordance with a selected confidence threshold received via a selection input to the graphical user interface.

An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing system, cause the computing system to perform operations comprising:

• obtaining a plurality of confidence thresholds for predictions made by a machine learning classifier, wherein the machine learning classifier is configured to:
• receive input observations
• predict, for each respective input observation, a set of probabilities that indicate a likelihood of each respective input observation belonging to each respective output category of a plurality of output categories;
• and
• classify each respective observation into the output category having the highest probability of the plurality of output categories;
• reclassifying, for each confidence threshold of a plurality of confidence thresholds, any of the input observations for which all probabilities of the set of probabilities are less than the confidence threshold into a null category that is not one of the plurality of output categories;
• after the reclassifying, determining, for each confidence threshold of the plurality of confidence thresholds and for each output category of the plurality of output categories, a precision value that indicates a percentage of input observations correctly predicted to be in a particular output category out of a total number of input observations predicted to be in the particular output category;
• after the reclassifying, determining, for each confidence threshold of the plurality of confidence thresholds and for each output category of the plurality of output categories, a coverage value that indicates a percentage of non-null-category predictions made for the input observations in the particular output category;
• after the reclassifying, determining, for each confidence threshold of the plurality of confidence thresholds and for each output category of the plurality of output categories, an objective function value, wherein the objective function value is a multiplicative product of the precision value and the coverage value;
• generating a graphical user interface for display, wherein the graphical user interface visually associates confidence thresholds, precision values, coverage values,
• and objective function values for each of the plurality of output categories, wherein the confidence threshold is one of a plurality of confidence thresholds on the graphical user interface, and wherein each of the plurality of confidence thresholds are selectably displayed with the associated precision values, coverage values, and objective function values;
• and
• performing, by at least the machine learning classifier, further classification of additional input observations into the output categories and the null category in accordance with a selected confidence threshold received via a selection input to the graphical user interface.