Back to results
US 11,900,236 B2Granted
Interpretable neural network
AssigneeUMNAI Limited
InventorsAngelo Dalli, Mauro Pirrone
Life of this patent
Estimating the term…
Abstract
An exemplary embodiment may provide an interpretable neural network with hierarchical conditions and partitions. A local function f(x) may model the feature attribution within a specific partition. The combination of all the local functions creates a globally interpretable model. Further, INNs may utilize an external process to identify suitable partitions during their initialization and may support training using back-propagation and related techniques.
Claims30
A system for providing an interpretable neural network embodied on a non-transitory computer readable medium, comprising:
- an input;
- a feature generation network configured to identify a plurality of features from the input;
- a plurality of relevance estimators, each relevance estimator configured to calculate a coefficient associated with one or more features of the input;
- a conditional network configured to evaluate a plurality of rules, each rule comprising at least an IF-condition, wherein each IF-condition activates one or more partitions, each partition comprising one or more features;
- a feature attribution layer provided downstream of the plurality of relevance estimators and configured to, based on the coefficient associated with the one or more features of the input, calculate an attribution value of the features associated with the partitions activated by the conditional network;
- an aggregation layer configured to aggregate a plurality of predictive results for each of the activated partitions, into a partition hierarchy comprising a plurality of hierarchical partitions, the hierarchical partitions comprising at least one pair of overlapping hierarchical partitions and at least one pair of non-overlapping hierarchical partitions, wherein the aggregation layer is configured to aggregate the plurality of predictive results by merging at least one pair of hierarchical partitions and splitting at least one hierarchical partition, wherein aggregation further comprises aggregating the plurality of relevance estimators;
- and
- an output layer configured to provide an output comprising an answer and explanation based on the predictive results and activated partitions;
- wherein the system is configured to simultaneously compute, in a single feed-forward step extending from the input to the output without iteration of intervening steps, the answer and explanation;
- wherein the answer comprises a response provided by the interpretable neural network to the input, and wherein the explanation comprises a set or sequence of decisions performed from the input to the output as part of the single feed-forward step.
7 more elements
A method for providing an interpretable neural network, comprising:
- inputting a set of training data to a black-box predictor model;
- recording an output of the black-box predictor model corresponding to the set of training data;
- with a feature attribution layer, receiving, from a plurality of relevance estimators, a plurality of coefficients associated with one or more features of the set of training data input to the black-box predictor model, and calculating one or more attribution values of the coefficients;
- with an aggregation layer, aggregating the output and forming one or more hierarchical partitions based on the aggregated output, the hierarchical partitions comprising at least one pair of overlapping hierarchical partitions and at least one pair of non-overlapping hierarchical partitions, wherein the aggregation layer is configured to aggregate the output by merging at least one pair of hierarchical partitions and splitting at least one hierarchical partition, and wherein aggregating the output further comprises aggregating the plurality of relevance estimators;
- applying at least one linear or non-linear transformation to the partitions to form one or more local models;
- constructing rules based on the local models;
- and
- aggregating the rules to form a global interpretable model configured to simultaneously compute, in a single feed-forward step extending from the input to the output without iteration of intervening steps, an answer and an explanation;
- wherein the answer comprises a response provided by the interpretable neural network to the input, and wherein the explanation comprises a set or sequence of decisions performed from the input to the output as part of the single feed-forward step.
6 more elements
Dependencies are inferred from the claim text.