Detecting poisoning attacks on neural networks by activation clustering

A system comprising:

• at least one processor;
• and
• a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:
• receiving a training set comprising a plurality of data points, wherein a neural network is trained as a classifier based on the training set;
• for each data point of the training set:
• classifying the data point with one of a plurality of classification labels using the trained neural network;
• and
• recording neuronal activations of a portion of the trained neural network in response to the data point;
• and
• for each classification label that a portion of the training set has been classified with:
• clustering a portion of all recorded neuronal activations that are in response to the portion of the training set;
• and
• detecting one or more poisonous data points in the portion of the training set based on the clustering.

A computer program product comprising a computer-readable hardware storage medium having program code embodied therewith, the program code being executable by a computer to implement a method comprising:

• receiving a training set comprising a plurality of data points, wherein a neural network is trained as a classifier based on the training set;
• for each data point of the training set:
• classifying the data point with one of a plurality of classification labels using the trained neural network;
• and
• recording neuronal activations of a portion of the trained neural network in response to the data point;
• and
• for each classification label that a portion of the training set has been classified with:
• clustering a portion of all recorded neuronal activations that are in response to the portion of the training set;
• and
• detecting one or more poisonous data points in the portion of the training set based on the clustering.