Detecting and mitigating poison attacks using data provenance

A computer-implemented method for provenance-based defense against poison attacks, the method comprising:

• receiving one or more observations from one or more data sources, wherein the one or more observations comprise a trusted dataset of data points and an untrusted dataset of data points, and wherein each observation comprises one or more features for training a final prediction model;
• receiving provenance data corresponding to each observation;
• determining whether some or all of the observations are poisoned based at least in part on the corresponding provenance data;
• and
• in response to determining some or all of the observations are poisoned, removing the poisoned observation(s) from a final training dataset used to train the final prediction model,
• determining a threshold performance difference indicative of poisoned observations, the determining comprising:
• randomly removing a subset of the untrusted dataset of data points from the untrusted dataset of data points to generate a first calibration dataset;
• randomly selecting a subset of the trusted dataset of data points to generate a second calibration dataset;
• training a prediction model corresponding to the final prediction model using the first calibration dataset;
• training the prediction model using the first and second calibration datasets;
• computing a difference in a performance of the prediction model when trained using the first calibration dataset against a performance of the prediction model when trained using the first and second calibration datasets;
• and
• setting the threshold performance difference to a value greater than or equal to the difference in the performance of the prediction model when trained using the first calibration dataset versus the performance of the prediction model when trained using the first and second calibration datasets.

A computer-implemented method for provenance-based defense against poison attacks in a fully untrusted data environment, the method comprising:

• receiving a dataset and associated provenance data, wherein the dataset comprises a plurality of untrusted data points and excludes trusted data points, wherein each untrusted data point is associated with one or more provenance signatures of the provenance data;
• randomly assigning a first portion of the untrusted data points to a training dataset;
• randomly assigning a second portion of the untrusted data points to a full evaluation dataset;
• grouping the untrusted data points of the training dataset into a plurality of groups each characterized by a different one of the provenance signatures, wherein untrusted data points of each group are characterized by a same one of the provenance signatures;
• for each group of the untrusted data points in the training dataset:
• training a supervised learning method using a first training dataset to generate a complete prediction model, wherein the first training dataset includes all of the untrusted data points;
• training the supervised learning method using a second training dataset to generate a filtered prediction model, wherein the second training dataset excludes the group of the untrusted data points;
• generating a new evaluation dataset by removing, from the full evaluation dataset, any data points sharing a provenance signature with one or more data points of the second training dataset;
• applying each of the complete prediction model and the filtered prediction model to the new evaluation dataset;
• comparing a performance of applying the complete prediction model to the new evaluation dataset to a performance of applying the filtered prediction model to the new evaluation dataset to determine whether the performance of applying the filtered prediction model to the new evaluation dataset exceeds the performance of applying the complete prediction model to the new evaluation dataset;
• in response to determining the performance of applying the filtered prediction model to the new evaluation dataset exceeds the performance of applying the complete prediction model to the new evaluation dataset:
• designating as poisonous one or more data points, in both the training dataset and the full evaluation dataset, the one or more data points having a same provenance signature as:
• the group of untrusted data points excluded from the second training dataset;
• and
• the data points removed from the full evaluation dataset based on sharing the provenance signature with the one or more data points of the second training dataset;
• and
• removing the one or more poisonous data points from the training dataset and the full evaluation dataset;
• and
• recombining the training dataset and full evaluation dataset after removing the one or more poisonous data points therefrom so as to create a final filtered training set;
• and
• training a final prediction model using the final filtered training set.