Abnormality detection based on causal graphs representing causal relationships of abnormalities

A method comprising:

• detecting, by a processor, an abnormality in a test data set via an abnormality detection model;
• generating, by the processor, one or more counterfactual data sets for the test data set;
• determining, by the processor, a quantitative feature dependence between the test data set and the one or more counterfactual data sets;
• determining, by the processor, a causal relationship of the abnormality based on the quantitative feature dependence via a causality discovery algorithm;
• generating, by the processor, a causal graph that represents the causal relationship of the abnormality,
• wherein the causality discovery algorithm is a pairwise causality discovery algorithm, and the determining of the causal relationship of the abnormality based on the quantitative feature dependence comprises:
• generating, via the abnormality detection model and based on the test data set with the abnormality, a predefined number of counterfactual data sets;
• evaluating pairwise causality, via the pairwise causality discovery algorithm, to recognize the causality relationship;
• and
• outputting the causality relationship, and
• wherein the abnormality detection model is trained via a second plurality of operations, the second plurality of operations comprising:
• generating feature weights based on information comprising feature labelling;
• training the abnormality detection model based on the feature weights and training data sets;
• removing abnormal data from the training data sets;
• and
• training, after removing the abnormal data from the training data sets, the abnormality detection model to be a counterfactual generation model that is configured to generate the one or more counterfactual data sets,
• wherein the determining the quantitative feature dependence between the test data set and the one or more counterfactual data sets comprises:
• calculating a difference in numerical value for each feature of abnormal data of the test data set and median values of the one or more counterfactual data sets;
• generating feature contributions of the abnormal data of the test data set and the one or more counterfactual data sets via a feature dependence interpretation model;
• generating feature contributions for each numerical difference of features of the abnormal data of the test data set;
• eliminating one or more feature dependence interpretations having feature contributions for a numerical difference of features of the abnormal data of the test data set that are less than a threshold;
• and
• transforming the feature contributions for a numerical difference of features of the abnormal data of the test data set that are not less than the threshold into a table that quantitatively describes feature dependence.

A non-transitory computer-readable storage device storing a plurality of instructions which, when executed by a processor, cause the processor to perform operations, the operations comprising:

• detecting an abnormality in a test data set via an abnormality detection model;
• generating one or more counterfactual data sets for the test data set;
• determining a quantitative feature dependence between the test data set and the one or more counterfactual data sets;
• determining a causal relationship of the abnormality based on the quantitative feature dependence via a causality discovery algorithm;
• and
• generating a causal graph that represents the causal relationship of the abnormality,
• wherein the causality discovery algorithm is a pairwise causality discovery algorithm, and the determining of the causal relationship of the abnormality based on the quantitative feature dependence comprises:
• generating, via the abnormality detection model and based on the test data set with the abnormality, a predefined number of counterfactual data sets;
• evaluating pairwise causality, via the pairwise causality discovery algorithm, to recognize the causality relationship;
• and
• outputting the causality relationship, and
• wherein the abnormality detection model is trained via a second plurality of operations, the second plurality of operations comprising:
• generating feature weights based on information comprising feature labelling;
• training the abnormality detection model based on the feature weights and training data sets;
• removing abnormal data from the training data sets;
• and
• training, after removing the abnormal data from the training data sets, the abnormality detection model to be a counterfactual generation model that is configured to generate the one or more counterfactual data sets,
• wherein the determining the quantitative feature dependence between the test data set and the one or more counterfactual data sets comprises;
• calculating a difference in numerical value for each feature of abnormal data of the test data set and median values of the one or more counterfactual data sets;
• generating feature contributions of the abnormal data of the test data set and the one or more counterfactual data sets via a feature dependence interpretation model;
• generating feature contributions for each numerical difference of features of the abnormal data of the test data set,
• eliminating one or more feature dependence interpretations having feature contributions for a numerical difference of features of the abnormal data of the test data set that are less than a threshold;
• and
• transforming the feature contributions for a numerical difference of features of the abnormal data of the test data set that are not less than the threshold into a table that quantitatively describes feature dependence.