# US 2021/0081718 A1 · Detecting Backdoor Attacks Using Exclusionary Reclassification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0081718 A1
- Document ID: US20210081718A1
- Status: Published application
- Kind: A1
- Application number: 16/571,318
- Filed: 2019-09-16
- Published: 2021-03-18
- Earliest claimed: 2019-09-16
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Nathalie Baracaldo Angel; Bryant Chen; Heiko H. Ludwig
- Web page: https://patentgrep.com/patent/US20210081718A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/2134: based on separation criteria, e.g. independent component analysis
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/23: Clustering techniques
- G06F 18/23211: with adaptive number of clusters
- G06F 18/24: Classification techniques
- G06F 18/24133: {Distances to prototypes}
- G06F 18/2433: Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7753: {Incorporation of unlabelled data, e.g. multiple instance learning [MIL]}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments relate to a system, program product, and method for processing an untrusted data set to automatically determine which data points there are poisonous. A neural network is trained network using potentially poisoned training data. Each of the training data points is classified using the network to retain the activations of at least one hidden layer, and segment those activations by the label of corresponding training data. Clustering is applied to the retained activations of each segment, and a clustering assessment is conducted to remove an identified cluster from the data set, form a new training set, and train a second neural model with the new training set. The removed cluster and corresponding data are applied to the trained second neural model to analyze and classify data in the removed cluster as either legitimate or poisonous.

## Claims

### Claim 1

A computer system comprising:
a processor operatively coupled to memory;
an artificial intelligence (AI) platform, in communication with the processing, having machine learning (ML) tools to process an untrusted training data set, the tools comprising:
a training manager to train a first neural model with the untrusted data set;
a ML manager, operatively coupled to the training manager, to classify each data point in the untrusted data set using the trained first neural model, and to retain activations of one or more designated layers in the trained first neural model;
a cluster manager, operatively coupled to the ML manager, to apply a clustering technique on the retained activations for each label, and for each cluster to assess integrity of data in the cluster, including the cluster manager to:
remove an identified cluster from the data set, and form a new training set with data remaining in the data set;
train a second neural model using the new training set; and
using the trained second neural model, analyze data in the removed cluster and assess alignment of the classified data with respect to a label assignment; and
a classification manager, operatively coupled to the cluster manager, the classification manager to assign a classification to the assessed cluster, the cluster classification corresponding to the alignment assessment, wherein the cluster classification is selected from the group consisting of: poisonous and legitimate.

### Claim 2 (depends on 1)

The system of claim 1, wherein alignment assessment of the classified data further comprises the cluster manager to:
compare data classification labels returned from the trained second neural model with an original data classification label; and
wherein the classification assignment is responsive to the data classification label comparison.

### Claim 3 (depends on 2)

The system of claim 2, wherein the comparison further comprises the cluster manager to identify a plurality of the returned labels matching the original classification label, and assign a legitimate classification to the cluster.

### Claim 4 (depends on 2)

The system of claim 2, wherein the comparison further comprises the cluster manager to identify a plurality of the returned labels conflicting with the original classification label, and assign a poisonous classification to the cluster.

### Claim 5 (depends on 2)

The system of claim 2, wherein the alignment of the classified data further comprises the cluster manager to compare a first value representative of quantity of data points classified by the trained second neural model with a second value representative of a quantity of the data points classified with a label representing a majority label, and wherein a classification assignment to the assessed cluster is responsive to the comparison.

### Claim 6 (depends on 5)

The system of claim 5, further comprising the cluster manager to assign a legitimate classification to the cluster when the comparison indicates that the first value is greater than the second value, and assign a poisonous classification to the cluster when the first value is less than the second value.

### Claim 7 (depends on 1)

The system of claim 1, further comprising a repair manager, operatively coupled to the cluster manager, and for each cluster classified as poisonous data, the repair manager to repair the cluster.

### Claim 8

A computer program product to utilize machine learning to process an untrusted training data set, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
train a first neural model with the untrusted data set;
classify each data point in the untrusted data set using the trained first neural model, and retain activations of one or more designated layers in the trained first neural model;
apply a clustering technique on the retained activations for each label, and for each cluster assess integrity of data in the cluster, including program code to:
remove an identified cluster from the data set, and form a new training set with data remaining in the data set;
train a second neural model using the new training set; and
using the trained second neural model, analyze data in the removed cluster and assess alignment of the classified data with respect to a label assignment; and
assign a classification to the assessed cluster, the cluster classification corresponding to the alignment assessment, wherein the cluster classification is selected from the group consisting of: poisonous and legitimate.

### Claim 9 (depends on 8)

The computer program product of claim 8, wherein alignment assessment of the classified data further comprises program code to:
compare data classification labels returned from the trained second neural model with an original data classification label; and
wherein the classification assignment is responsive to the data classification label comparison.

### Claim 10 (depends on 9)

The computer program product of claim 9, wherein the comparison further comprises program code to identify a plurality of the returned labels matching the original classification label, and assign a legitimate classification to the cluster.

### Claim 11 (depends on 9)

The computer program product of claim 9, wherein the comparison further comprises program code to identify a plurality of the returned labels conflicting with the original classification label, and assign a poisonous classification to the cluster.

### Claim 12 (depends on 9)

The computer program product of claim 9, wherein the alignment of the classified data further comprises program code to compare a first value representative of a quantity of data points classified by the trained second neural model with a second value representative of a quantity of the data points classified with a label representing a majority label, and wherein a classification assignment to the assessed cluster is responsive to the comparison.

### Claim 13 (depends on 12)

The computer program product of claim 12, further comprising program code to assign a legitimate classification to the cluster when the comparison indicates that the first value is greater than the second value, and assign a poisonous classification to the cluster when the first value is less than the second value.

### Claim 14 (depends on 8)

The computer program product of claim 8, further comprising program code to repair each cluster classified as poisonous data.

### Claim 15

A method comprising:
receiving, by a neural network, an untrusted training data set, each data point of the untrusted data set having a label;
training a first neural model using the untrusted data set;
classifying each data point in the untrusted data set using the trained first neural model, and retaining activations of one or more designated layers in the trained first neural model;
applying a clustering technique on the retained activations for each label;
for each cluster, assessing integrity of data in the cluster including:
removing an identified cluster from the data set, and forming a new training set with data remaining in the data set;
training a second neural model using the new training set; and
using the trained second neural model, analyzing data in the removed cluster and assessing alignment of the classified data with respect to a label assignment; and
assigning a classification to the assessed cluster, the cluster classification corresponding to the alignment assessment, wherein the cluster classification is selected from the group consisting of: poisonous and legitimate.

### Claim 16 (depends on 15)

The method of claim 15, wherein assessing alignment of the classified data further comprises:
comparing data classification labels returned from the trained second neural model with an original data classification label; and
wherein the classification assignment is responsive to the data classification label comparison.

### Claim 17 (depends on 16)

The method of claim 16, further comprising the comparison identifying a plurality of the returned labels matching the original classification label, and assigning a legitimate classification to the cluster.

### Claim 18 (depends on 16)

The method of claim 16, further comprising the comparison identifying a plurality of the returned labels conflicting with the original classification label, and assigning a poisonous classification to the cluster.

### Claim 19 (depends on 16)

The method of claim 16, wherein assessing alignment of the classified data further comprises comparing a first value representative a quantity of data points classified by the trained second neural model with a second value representative of a quantity of the data points classified with a label representing a majority label, and wherein assigning a classification to the assessed cluster is responsive to the comparison.

### Claim 20 (depends on 19)

The method of claim 19, further comprising assigning a legitimate classification to the cluster when the comparison indicates that the first value is greater than the second value, and assign a poisonous classification to the cluster when the first value is less than the second value, wherein the poisonous cluster is subject to an action selected from the group consisting of: repair and removal.
