# US 12,596,941 B2 · Realistic counterfactual explanation of machine learning predictions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,596,941 B2 - Document ID: US12596941B2 - Status: Granted - Kind: B2 - Application number: 17/378,794 - Filed: 2021-07-19 - Published: 2023-01-26 - Granted: 2026-04-07 - Earliest claimed: 2021-07-19 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Michael Vinov; Oleg Blinder; Diptikalyan Saha; Sandeep Hans; Aniya Aggarwal; Omer Yehuda Boehm; Eyal Bin - Web page: https://patentgrep.com/patent/US12596941B2 ## CPC - G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer-implemented method comprising, automatically: analyzing a machine learning dataset which comprises multiple datapoints, to deduce constraints on features of the datapoints; generating a first set of CSP (Constraint Satisfaction Problem) rules expressing the constraints; based on a machine learning model which was trained on the dataset, generating a second set of CSP rules that define one or more perturbation candidates among the features of one of the datapoints; formulating a CSP based on the first and second sets of CSP rules; solving the formulated CSP using a solver; and using the solution of the CSP as a counterfactual explanation of a prediction made by the machine learning model with respect to the one datapoint. ## Claims ### Claim 1 A computer-implemented method comprising: automatically analyzing a machine learning dataset comprising at least one datapoint to deduce constraints on features of the at least one datapoint; generating a first set of CSP (Constraint Satisfaction Problem) rules expressing the constraints; based on a machine learning model trained on the machine learning dataset, generating a second set of CSP rules that define one or more perturbation candidates among the features of the at least one datapoint, wherein the perturbation candidates are one or more changes to features associated with the at least one datapoint, comprising: generating a modified datapoint based on changed feature values associated with the perturbation candidates and the at least one datapoint; formulating a CSP based on the first and second sets of CSP rules; solving the formulated CSP using a solver; using the solution of the CSP as a counterfactual explanation of a prediction made by the machine learning model with respect to the at least one datapoint, wherein the counterfactual explanation is a natural language representation of the solution of the CSP; and responsive to an accessible loss function of the machine learning model, iterating until an opposite classification is achieved, and subsequently using a set of constraints as the second set of CSP rules, wherein, in each iteration, the modified datapoint of a previous iteration serves as the at least one datapoint of a current iteration. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the analyzing of the machine learning dataset comprises analyzing values of the features of the at least one datapoint. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the analyzing of the values of the features is to discover descriptive statistics of the values across the machine learning dataset. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the current iteration comprises: modifying the loss function of the machine learning model to use a flipped class label; calculating gradients of the modified loss function with respect to the at least one datapoint; translating the calculated gradients into the set of constraints, and formulating another CSP based on the set of constraints and on the first set of CSP rules; solving the other CSP using the solver, to identify the one or more perturbation candidates among the features of the at least one datapoint; using the machine learning model to predict a class for the modified datapoint. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the generating of the second set of CSP comprises, when a loss function of the machine learning model is inaccessible: calculating an anchor for the at least one datapoint using the machine learning model, wherein the anchor is a definition of conditions that are a likely cause of a certain classification by the machine learning model; and translating the conditions in the second set of CSP rules. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the at least one datapoint is randomly selected among all datapoints in the machine learning dataset. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, executed by at least one hardware processor. ### Claim 8 A system comprising: (i) at least one hardware processor; and (ii) a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by said at least one hardware processor to: automatically analyze a machine learning dataset comprising at least one datapoint to deduce constraints on features of the at least one datapoint; generate a first set of CSP (Constraint Satisfaction Problem) rules expressing the constraints; based on a machine learning model trained on the machine learning dataset, generate a second set of CSP rules that define one or more perturbation candidates among the features at least one datapoint in the one or more datapoints, wherein the perturbation candidates are one or more changes to features associated with the at least one datapoint, comprising: generating a modified datapoint based on changed feature values associated with the perturbation candidates and the at least one datapoint; formulate a CSP based on the first and second sets of CSP rules; solve the formulated CSP using a solver; use the solution of the CSP as a counterfactual explanation of a prediction made by the machine learning model with respect to the at least one datapoint, wherein the counterfactual explanation is a natural language representation of the solution of the CSP; and responsive to an accessible loss function of the machine learning model, iterate until an opposite classification is achieved, and subsequently using a set of constraints as the second set of CSP rules, wherein, in each iteration, the modified datapoint of a previous iteration serves as the at least one datapoint of a current iteration. ### Claim 9 (depends on 8) The system of claim 8, wherein the analyzing of the machine learning dataset comprises analyzing values of the features of the at least one datapoint. ### Claim 10 (depends on 9) The system of claim 9, wherein the analyzing of the values of the features is to discover descriptive statistics of the values across the machine learning dataset. ### Claim 11 (depends on 8) The system of claim 8, wherein the current iteration comprises: modifying the loss function of the machine learning model to use a flipped class label; calculating gradients of the modified loss function with respect to the at least one datapoint; translating the calculated gradients into the set of constraints, and formulating another CSP based on the set of constraints and on the first set of CSP rules; solving the other CSP using the solver, to identify the one or more perturbation candidates among the features of the at least one datapoint; using the machine learning model to predict a class for the modified datapoint. ### Claim 12 (depends on 8) The system of claim 8, wherein the generating of the second set of CSP comprises, when a loss function of the machine learning model is inaccessible: calculating an anchor for the at least one datapoint using the machine learning model, wherein the anchor is a definition of conditions that are a likely cause of a certain classification by the machine learning model; and translating the conditions in the second set of CSP rules. ### Claim 13 (depends on 8) The system of claim 8, wherein the at least one datapoint is randomly selected among all datapoints in the dataset. ### Claim 14 A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therewith, the program code executable by at least one hardware processor to: automatically analyze a machine learning dataset comprising at least one datapoints to deduce constraints on features of the at least one datapoint; generate a first set of CSP (Constraint Satisfaction Problem) rules expressing the constraints; based on a machine learning model trained on the machine learning dataset, generate a second set of CSP rules that define one or more perturbation candidates among the features at least one datapoint, wherein the perturbation candidates are one or more changes to features associated with the at least one datapoint, comprising: generating a modified datapoint based on changed feature values associated with the perturbation candidates and the at least one datapoint; formulate a CSP based on the first and second sets of CSP rules; solve the formulated CSP using a solver; use the solution of the CSP as a counterfactual explanation of a prediction made by the machine learning model with respect to the at least one datapoint, wherein the counterfactual explanation is a natural language representation of the solution of the CSP; and responsive to an accessible loss function of the machine learning model, iterate until an opposite classification is achieved, and subsequently using a set of constraints as the second set of CSP rules, wherein, in each iteration, the modified datapoint of a previous iteration serves as the at least one datapoint of a current iteration. ### Claim 15 (depends on 14) The computer program product of claim 14, wherein the analyzing of the machine learning dataset comprises analyzing values of the features of the at least one datapoint. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the analyzing of the values of the features is to discover descriptive statistics of the values across the machine learning dataset. ### Claim 17 (depends on 14) The computer program product of claim 14, wherein current iteration comprises: modifying the loss function of the machine learning model to use a flipped class label; calculating gradients of the modified loss function with respect to the at least one datapoint; translating the calculated gradients into the set of constraints, and formulating another CSP based on the set of constraints and on the first set of CSP rules; solving the other CSP using the solver, to identify the one or more perturbation candidates among the features of the at least one datapoint; using the machine learning model to predict a class for the modified datapoint. ### Claim 18 (depends on 14) The computer program product of claim 14, wherein the generating of the second set of CSP comprises, when a loss function of the machine learning model is inaccessible: calculating an anchor for the at least one datapoint using the machine learning model, wherein the anchor is a definition of conditions that are a likely cause of a certain classification by the machine learning model; and translating the conditions in the second set of CSP rules. ### Claim 19 (depends on 14) The computer program product of claim 14, wherein the one datapoint is randomly selected among all datapoints in the machine learning dataset.