# US 2019/0318099 A1 · Using Gradients to Detect Backdoors in Neural Networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318099 A1
- Document ID: US20190318099A1
- Status: Published application
- Kind: A1
- Application number: 15/953,956
- Filed: 2018-04-16
- Published: 2019-10-17
- Earliest claimed: 2018-04-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Wilka Carvalho; Bryant Chen; Benjamin J. Edwards; Taesung Lee; Ian M. Molloy; Jialong Zhang
- Web page: https://patentgrep.com/patent/US20190318099A1

## CPC

- G06F 21/577: {Assessing vulnerabilities and evaluating computer system security}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 20/00: Machine learning
- G06F 2221/033: Test or assess software
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Mechanisms are provided for evaluating a trained machine learning model to determine whether the machine learning model has a backdoor trigger. The mechanisms process a test dataset to generate output classifications for the test dataset, and generate, for the test dataset, gradient data indicating a degree of change of elements within the test dataset based on the output generated by processing the test dataset. The mechanisms analyze the gradient data to identify a pattern of elements within the test dataset indicative of a backdoor trigger. The mechanisms generate, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model.

## Claims

### Claim 1

A method, in a data processing system comprising a processor and a memory, the memory comprising instructions which are executed by the processor to specifically configure the processor to implement a backdoor trigger identification engine, for evaluating a trained machine learning model to determine whether the machine learning model has a backdoor trigger, the method comprising:
processing, by the trained machine learning model, a test dataset to generate output classifications for the test dataset;
generating, by the backdoor trigger identification engine, for the test dataset, gradient data indicating a degree of change of elements within the test dataset based on the output generated by processing the test dataset;
analyzing, by the backdoor trigger identification engine, the gradient data to identify a pattern of elements within the test dataset indicative of a backdoor trigger; and
generating, by the backdoor trigger identification engine, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the test dataset comprises a plurality of digital images, and wherein a sub-portion of the test dataset comprises one or more digital images having a portion of the digital images that comprises a characteristic triggering a misclassification, by the trained machine learning model, of at least one object in the digital image into an incorrect class in the output classifications.

### Claim 3 (depends on 2)

The method of claim 2, wherein the portion of the digital images is one of a single pixel or a pattern of pixels having at least one characteristic that causes the misclassification of the at least one object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the gradient data is generated by computing a gradient of a loss function of the trained machine learning model with respect to each component of the test dataset.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating, for the test dataset, gradient data indicating a degree of change of elements within the test dataset further comprises introducing noise into the test dataset to mask features causing legitimate changes in the gradient data.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
retraining the machine learning model based on the identified backdoor trigger to cause the machine learning model to correctly classify portions of datasets that include the backdoor trigger into a correct classification.

### Claim 7 (depends on 6)

The method of claim 6, wherein retraining the machine learning model comprises training the machine learning model on a new training data set with the backdoor trigger inserted into the new training data set.

### Claim 8 (depends on 1)

The method of claim 1, wherein analyzing, by the backdoor trigger identification engine, the gradient data to identify a pattern of elements indicative of a backdoor trigger comprises automatically analyzing, by an anomaly detection algorithm or strength comparison algorithm, the gradient data to identify the pattern of elements indicative of the backdoor trigger.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model comprises at least one of logging the existence of the backdoor trigger in an entry of a log data structure or sending a notification message to an authorized user, wherein either the log data structure or notification message comprises an identity of the backdoor trigger, the output classifications associated with the backdoor trigger, and characteristics of the backdoor trigger.

### Claim 10 (depends on 1)

The method of claim 1, wherein the machine learning model is a convolutional neural network.

### Claim 11

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a backdoor trigger identification engine, for evaluating a trained machine learning model to determine whether the machine learning model has a backdoor trigger, wherein the data processing system operates to:
process, by the trained machine learning model, a test dataset to generate output classifications for the test dataset;
generate, by the backdoor trigger identification engine, for the test dataset, gradient data indicating a degree of change of elements within the test dataset based on the output generated by processing the test dataset;
analyze, by the backdoor trigger identification engine, the gradient data to identify a pattern of elements within the test dataset indicative of a backdoor trigger; and
generate, by the backdoor trigger identification engine, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein the test dataset comprises a plurality of digital images, and wherein a sub-portion of the test dataset comprises one or more digital images having a portion of the digital images that comprises a characteristic triggering a misclassification, by the trained machine learning model, of at least one object in the digital image into an incorrect class in the output classifications.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein the portion of the digital images is one of a single pixel or a pattern of pixels having at least one characteristic that causes the misclassification of the at least one object.

### Claim 14 (depends on 11)

The computer program product of claim 11, wherein the gradient data is generated at least by computing a gradient of a loss function of the trained machine learning model with respect to each component of the test dataset.

### Claim 15 (depends on 11)

The computer program product of claim 11, wherein the computer readable program further causes the data processing system to generate, for the test dataset, gradient data indicating a degree of change of elements within the test dataset further at least by introducing noise into the test dataset to mask features causing legitimate changes in the gradient data.

### Claim 16 (depends on 11)

The computer program product of claim 11, wherein the computer readable program further causes the data processing system to:
retrain the machine learning model based on the identified backdoor trigger to cause the machine learning model to correctly classify portions of datasets that include the backdoor trigger into a correct classification.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein retraining the machine learning model comprises training the machine learning model on a new training data set with the backdoor trigger inserted into the new training data set.

### Claim 18 (depends on 11)

The computer program product of claim 11, wherein the computer readable program further causes the data processing system to analyze, by the backdoor trigger identification engine, the gradient data to identify a pattern of elements indicative of a backdoor trigger at least by automatically analyzing, by an anomaly detection algorithm or strength comparison algorithm, the gradient data to identify the pattern of elements indicative of the backdoor trigger.

### Claim 19 (depends on 11)

The computer program product of claim 11, wherein the computer readable program further causes the data processing system to generate, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model comprising at least one of logging the existence of the backdoor trigger in an entry of a log data structure or sending a notification message to an authorized user, wherein either the log data structure or notification message comprises an identity of the backdoor trigger, the output classifications associated with the backdoor trigger, and characteristics of the backdoor trigger.

### Claim 20

An apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement a backdoor trigger identification engine, for evaluating a trained machine learning model to determine whether the machine learning model has a backdoor trigger, wherein the processor operates to:
process, by the trained machine learning model, a test dataset to generate output classifications for the test dataset;
generate, by the backdoor trigger identification engine, for the test dataset, gradient data indicating a degree of change of elements within the test dataset based on the output generated by processing the test dataset;
analyze, by the backdoor trigger identification engine, the gradient data to identify a pattern of elements within the test dataset indicative of a backdoor trigger; and
generate, by the backdoor trigger identification engine, in response to the analysis identifying the pattern of elements indicative of a backdoor trigger, an output indicating the existence of the backdoor trigger in the trained machine learning model.
