# US 11,443,045 B2 · Methods and systems for explaining a decision process of a machine learning model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,045 B2 - Document ID: US11443045B2 - Status: Granted - Kind: B2 - Application number: 16/866,997 - Filed: 2020-05-05 - Published: 2021-11-11 - Granted: 2022-09-13 - Earliest claimed: 2020-05-05 - Assignees: Booz Allen Hamilton Inc. - Applicants: Booz Allen Hamilton Inc. - Inventors: Arash Rahnama-Moghaddam; Andrew Tseng - Web page: https://patentgrep.com/patent/US11443045B2 ## CPC - G06F 21/552: {involving long-term monitoring or reporting} - G06F 21/577: {Assessing vulnerabilities and evaluating computer system security} - G06F 16/2358: {Change logging, detection, and notification (replication G06F16/27)} - G06N 20/00: Machine learning - G06F 2221/034: Test or assess a computer or a system - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method and system for explaining a decision process of a machine learning model that includes inputting into a machine learning model a first input data file; receiving a first output data file from the machine learning model based on the first input data file; executing an adversarial attack on the machine learning model, creating a mapping of the one or more units of data of the first input data file with changes by the adversarial attack exceeding a first threshold to one or more segments of the first input data file; determining a density of the changes to the one or more units of data in each of the one or more segments; and displaying the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface. ## Claims ### Claim 1 A method for explaining a decision process of a machine learning model, the method comprising: inputting into a machine learning model a first input data file, the first input data file having one or more units of data; receiving a first output data file from the machine learning model based on the first input data file, wherein the first output data file has a first output determined by a decision process of the machine learning model; executing an adversarial attack on the machine learning model, the adversarial attack generating one or more second input data files and inputting the one or more second input data files into the machine learning model to produce one or more second output data files, wherein the adversarial attack changes one or more units of data of the first input data file to generate the one more second input data files; creating a first mapping of the one or more units of data of the first input data file changed by the adversarial attack to generate the one more second input data files; determining whether the changes to the one or more units of data of the first input data file to generate the one more second input data files exceed a first threshold, the first threshold based on the intensity of changes to the one or more units of data of the first input data file; segmenting the one or more units of data of the first input data file into one or more segments based on one or more characteristics of the first input data file; creating a second mapping of the one or more units of data of the first input data file with changes exceeding the first threshold to the one or more segments of the first input data file; determining a density of the changes to the one or more units of data in each of the one or more segments of the first input data file in the second mapping; and displaying the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface. ### Claim 2 (depends on 1) A method as in claim 1 wherein the machine learning model is a trained computer vision model. ### Claim 3 (depends on 1) The method as in claim 1, wherein the first input data file is an image file and the one or more units of data of the first input data file is one or more pixels. ### Claim 4 (depends on 1) The method as in claim 1, wherein the adversarial attack is a fast gradient sign method (FGSM) attack. ### Claim 5 (depends on 1) The method as in claim 1, wherein the one or more segments displayed represent the one or more segments of the first input data file used by the decision process of the machine learning model to generate the first output data file. ### Claim 6 A system for identifying the decision process of a machine learning model, the system comprising: a processor configured to input into a machine learning model a first input data file, the first input data file having one or more units of data; the processor configured to receive a first output data file from the machine learning model based on the first input data file, wherein the first output data file has a first output determined by a decision process of the machine learning model; an attack module configured to execute an adversarial attack on the machine learning model, the adversarial attack generating one or more second input data files and inputting the one or more second input data files into the machine learning model to produce one or more second output data files, wherein the adversarial attack changes one or more units of data of the first input data file to generate the one more second input data files; a first mapping module configured to create a first mapping of the one or more units of data of the first input data file changed by the adversarial attack to generate the one more second input data files; an attack determination module configured to determine whether the changes to the one or more units of data of the first input data file to generate the one more second input data files exceed a first threshold, the first threshold based on the intensity of changes to the one or more units of data of the first input data file; a segmentation module configured to segment the one or more units of data of the first input data file into one or more segments based on one or more characteristics of the first input data file; a second mapping module configured to create a second mapping of the one or more units of data of the first input data file with changes exceeding the first threshold to the one or more segments of the first input data file; a density determination module configured to determine a density of the changes to the one or more units of data in each of the one or more segments of the first input data file in the second mapping; and a display module configured to display the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface. ### Claim 7 (depends on 6) The system as in claim 6, wherein the machine learning model is a trained computer vision model. ### Claim 8 (depends on 6) The system as in claim 6, wherein the first input data file is an image file and the one or more units of data of the first input data file is one or more pixels. ### Claim 9 (depends on 6) The system as in claim 6, wherein the adversarial attack is a fast gradient sign method (FGSM) attack. ### Claim 10 (depends on 6) The system as in claim 6, wherein the one or more segments displayed represent the one or more segments of the first input data file used by the decision process of the machine learning model to generate the first output data file. ### Claim 11 A computer program product for identifying the decision process of a machine learning model, the computer program product comprising: a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method, comprising: inputting into a machine learning model a first input data file, the first input data file having one or more units of data; receiving a first output data file from the machine learning model based on the first input data file, wherein the first output data file has a first output determined by a decision process of the machine learning model; executing an adversarial attack on the machine learning model, the adversarial attack generating one or more second input data files and inputting the one or more second input data files into the machine learning model to produce one or more second output data files, wherein the adversarial attack changes one or more units of data of the first input data file to generate the one more second input data files; creating a first mapping of the one or more units of data of the first input data file changed by the adversarial attack to generate the one more second input data files; determining whether the changes to the one or more units of data of the first input data file to generate the one more second input data files exceed a first threshold, the first threshold based on the intensity of changes to the one or more units of data of the first input data file; segmenting the one or more units of data of the first input data file into one or more segments based on one or more characteristics of the first input data file; creating a second mapping of the one or more units of data of the first input data file with changes exceeding the first threshold to the one or more segments of the first input data file; determining a density of the changes to the one or more units of data in each of the one or more segments of the first input data file in the second mapping; and displaying the one or more segments of the first input data file having a density of changes to the one or more units of data exceeding a second threshold via a graphical user interface. ### Claim 12 (depends on 11) The computer program product as in claim 11, wherein the machine learning model is a trained computer vision model. ### Claim 13 (depends on 11) The computer program product as in claim 11, wherein the first input data file is an image file and the one or more units of data of the first input data file is one or more pixels. ### Claim 14 (depends on 11) The computer program product as in claim 11, wherein the adversarial attack is a fast gradient sign method (FGSM) attack. ### Claim 15 (depends on 13) The computer program product as in claim 13, wherein the one or more segments displayed represent the one or more segments of the first input data file used by the decision process of the machine learning model to generate the first output data file.