# US 11,907,334 B2 · Neural network negative rule extraction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,334 B2 - Document ID: US11907334B2 - Status: Granted - Kind: B2 - Application number: 17/115,610 - Filed: 2020-12-08 - Published: 2022-06-09 - Granted: 2024-02-20 - Earliest claimed: 2020-12-08 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Franck Vinh Le; Mudhakar Srivatsa - Web page: https://patentgrep.com/patent/US11907334B2 ## CPC - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06N 3/02: Neural networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 5/025: Extracting rules from data - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A first classification is received from a neural network regarding a training dataset sent to the neural network. A modified training dataset with a perturbation of the training dataset is identified, where this modified training dataset causes the neural network to return a second classification. The perturbation is analyzed to identify a negative rule of the neural network. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a first classification from a neural network regarding a training dataset sent to the neural network, wherein the neural network is configured to identify positive and negative evidence; identifying a modified training dataset with a perturbation of the training dataset that causes the neural network to return a second classification that is different from the first classification; and analyzing the perturbation to identify a negative rule of the neural network, wherein the negative rule relates to an absence of an element in an input of the training dataset increasing a likelihood that the neural network returns the first classification, wherein a presence of the element increases a likelihood that the neural network returns the second classification. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying features that are present in the modified training dataset that are not in the training dataset. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying that some of the features that are not in the training dataset are more relevant to the training dataset. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein identifying that some of the features that are not in the training dataset are more relevant includes using term frequency inverse document frequency. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying features of the training dataset that were most impacted by the perturbation. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the perturbation is determined using an adversarial attack method. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the perturbation is determined using a gradient-free optimization technique. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising interpreting output from the neural network using the negative rule. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, further comprising identifying the perturbation as a minimal perturbation to cause the neural network to return the second classification. ### Claim 10 A system comprising: a processor; and a memory in communication with the processor, the memory containing instructions that, when executed by the processor, cause the processor to: receive a first classification from a neural network regarding a training dataset sent to the neural network, wherein the neural network is configured to identify positive and negative evidence; identify a modified training dataset with a perturbation of the training dataset that causes the neural network to return a second classification that is different from the first classification; and analyze the perturbation to identify a negative rule of the neural network, wherein the negative rule relates to an absence of an element in an input of the training dataset increasing a likelihood that the neural network returns the first classification, wherein a presence of the element increases a likelihood that the neural network returns the second classification. ### Claim 11 (depends on 10) The system of claim 10, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying features that are present in the modified training dataset that are not in the training dataset. ### Claim 12 (depends on 11) The system of claim 11, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying that some of the features that are not in the training dataset are more relevant to the training dataset using term frequency inverse document frequency. ### Claim 13 (depends on 10) The system of claim 10, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying features of the training dataset that were most impacted by the perturbation to create the modified training dataset. ### Claim 14 (depends on 10) The system of claim 10, wherein the perturbation is determined using an adversarial attack method. ### Claim 15 (depends on 10) The system of claim 10, the memory containing additional instructions that, when executed by the processor, cause the processor to interpret output from the neural network using the negative rule. ### Claim 16 A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to: receive a first classification from a neural network regarding a training dataset sent to the neural network, wherein the neural network is configured to identify positive and negative evidence; identify a modified training dataset with a perturbation of the training dataset that causes the neural network to return a second classification that is different from the first classification; and analyze the perturbation to identify a negative rule of the neural network, wherein the negative rule relates to an absence of an element in an input of the training dataset increasing a likelihood that the neural network returns the first classification, wherein a presence of the element increases a likelihood that the neural network returns the second classification. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein analyzing the perturbation to identify the negative rule of the neural network includes at least one of: identifying features that are present in the modified training dataset that are not in the training dataset; and identifying that some of the features that are not in the training dataset are more relevant to the training dataset using term frequency inverse document frequency. ### Claim 18 (depends on 16) The computer program product of claim 16, wherein analyzing the perturbation to identify the negative rule of the neural network includes identifying features of the training dataset that were most impacted by the perturbation to create the modified training dataset. ### Claim 19 (depends on 16) The computer program product of claim 16, wherein the perturbation is determined using an adversarial attack method. ### Claim 20 (depends on 16) The computer program product of claim 16, the computer readable storage medium having additional program instructions embodied therewith that are executable by the computer to cause the computer to interpret output from the neural network using the negative rule.