Detection and mitigation of machine learning model adversarial attacks
AssigneeCitibank, N.A.
InventorsWilliam Franklin Cameron, Pramod Goyal, Prithvi Narayana Rao, Manjit Rajaretnam, Miriam Silver
Life of this patent
Estimating the term…
Abstract
Described herein are systems and methods for verifying the integrity of data, such as data used for training machine learning models. Some implementations are directed to verifying the provenance of datasets, the contents of datasets, or both. In some implementations, multiple filters are selected for verifying the contents of datasets. Filters can be selected based on rules, random selection, or using a machine learning model in some implementations. In some implementations, data cleaning is provided.
Claims20
A method for training a machine learning model, the method comprising:
- accessing a candidate training dataset for the machine learning model;
- evaluating the candidate training dataset using a first filter layer, wherein evaluating the candidate training dataset using the first filter layer comprises:
- applying at least one provenance filter to verify a provenance of the candidate training dataset by verifying at least one of:
- a signature of the candidate training dataset, a hash of the candidate training dataset, or a secure sockets layer/transport layer security (SSL/TLS) certificate of a source of the candidate training dataset;
- and
- when the provenance of the candidate training dataset is verified:
- evaluating the candidate training dataset using a second filter layer, wherein evaluating the candidate training dataset using the second filter layer comprises:
- determining at least one content filter to apply to the candidate training dataset, wherein the at least one content filter is configured to determine whether the candidate training dataset contains poisoned data;
- applying the at least one content filter to the candidate training dataset;
- determining, based on applying the at least one content filter to the candidate training dataset, an integrity level of the candidate training dataset;
- determining whether the integrity level satisfies a threshold integrity value;
- and
- when the integrity level satisfies the threshold integrity value, training the machine learning model using the candidate training dataset.
A method for training a machine learning model, the method comprising:
- accessing a candidate training dataset for the machine learning model;
- evaluating the candidate training dataset using a filter layer, wherein evaluating the candidate training dataset using the filter layer comprises:
- determining at least one content filter to apply to the candidate training dataset, wherein the at least one content filter is configured to determine if the candidate training dataset contains poisoned data;
- applying the at least one content filter to the candidate training dataset;
- and
- determining, based on applying the at least one content filter to the candidate training dataset, an integrity level of the candidate training dataset;
- determining if the integrity level satisfies a threshold integrity value;
- and
- when the integrity level satisfies the threshold integrity value, training the machine learning model using the candidate training dataset.
A method for cleaning training data, the method comprising:
- accessing a candidate training dataset for a machine learning model;
- evaluating the candidate training dataset using a filter layer, wherein evaluating the candidate training dataset using the filter layer comprises:
- determining at least one content filter to apply to the candidate training dataset, wherein the at least one content filter is configured to determine if the candidate training dataset contains poisoned data;
- applying the at least one content filter to the candidate training dataset;
- and
- determining, based on applying the at least one content filter to the candidate training dataset, an integrity level of the candidate training dataset;
- determining that the integrity level does not satisfy a threshold integrity value;
- identifying at least one poisoned data record in the candidate training dataset;
- and
- removing the at least one poisoned data record from the candidate training dataset to generate a cleaned training dataset.
Dependencies are inferred from the claim text.