Human-in-the-loop evaluation for explainable artificial intelligence

A system, comprising:

• a processor configured to:
• receive a data stream;
• analyze data elements of the data stream using one or more machine learning models and one or more machine learning prediction explanation implementations;
• test different candidate presentations associated with machine learning results provided to different reviewers in a group of human-in-the-loop reviewers that review predictions of the one or more machine learning models, wherein the different candidate presentations include different explanations generated by the one or more machine learning prediction explanation implementations and at least one control candidate presentation corresponding to an absent explanation;
• monitor different aspects of the testing;
• and
• use results of the monitoring to make a selection among the different candidate presentations;
• and
• a memory coupled to the processor and configured to provide the processor with instructions.

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

• receiving a data stream;
• analyzing data elements of the data stream using one or more machine learning models and one or more machine learning prediction explanation implementations;
• testing different candidate presentations associated with machine learning results provided to different reviewers in a group of human-in-the-loop reviewers that review predictions of the one or more machine learning models, wherein the different candidate presentations include different explanations generated by the one or more machine learning prediction explanation implementations and at least one control candidate presentation corresponding to an absent explanation;
• monitoring different aspects of the testing;
• and
• using results of the monitoring to make a selection among the different candidate presentations.