Methods and systems for explaining a decision process of a machine learning model

A system for identifying the decision process of a machine learning model, the system comprising:

• a processor configured to input into a machine learning model a first input data file, the first input data file having one or more units of data;
• the processor configured to receive a first output data file from the machine learning model based on the first input data file, wherein the first output data file has a first output determined by a decision process of the machine learning model;
• an attack module configured to execute an adversarial attack on the machine learning model, the adversarial attack generating one or more second input data files and inputting the one or more second input data files into the machine learning model to produce one or more second output data files, wherein the adversarial attack changes one or more units of data of the first input data file to generate the one more second input data files;
• a first mapping module configured to create a first mapping of the one or more units of data of the first input data file changed by the adversarial attack to generate the one more second input data files;
• an attack determination module configured to determine whether the changes to the one or more units of data of the first input data file to generate the one more second input data files exceed a first threshold, the first threshold based on the intensity of changes to the one or more units of data of the first input data file;
• a segmentation module configured to segment the one or more units of data of the first input data file into one or more segments based on one or more characteristics of the first input data file;
• a second mapping module configured to create a second mapping of the one or more units of data of the first input data file with changes exceeding the first threshold to the one or more segments of the first input data file;
• a density determination module configured to determine a density of the changes to the one or more units of data in each of the one or more segments of the first input data file in the second mapping;
• and
• a display module configured to display the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface.

A computer program product for identifying the decision process of a machine learning model, the computer program product comprising:

• a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method, comprising:
• inputting into a machine learning model a first input data file, the first input data file having one or more units of data;
• receiving a first output data file from the machine learning model based on the first input data file, wherein the first output data file has a first output determined by a decision process of the machine learning model;
• executing an adversarial attack on the machine learning model, the adversarial attack generating one or more second input data files and inputting the one or more second input data files into the machine learning model to produce one or more second output data files, wherein the adversarial attack changes one or more units of data of the first input data file to generate the one more second input data files;
• creating a first mapping of the one or more units of data of the first input data file changed by the adversarial attack to generate the one more second input data files;
• determining whether the changes to the one or more units of data of the first input data file to generate the one more second input data files exceed a first threshold, the first threshold based on the intensity of changes to the one or more units of data of the first input data file;
• segmenting the one or more units of data of the first input data file into one or more segments based on one or more characteristics of the first input data file;
• creating a second mapping of the one or more units of data of the first input data file with changes exceeding the first threshold to the one or more segments of the first input data file;
• determining a density of the changes to the one or more units of data in each of the one or more segments of the first input data file in the second mapping;
• and
• displaying the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface.