System and method for detecting and preventing model inversion attacks
AssigneeMicrosoft Technology Licensing, LLC
InventorsDushyant Sharma, Patrick Aubrey Naylor, William Francis Ganong, III, Uwe Helmut Jost, Ljubomir Milanovic
Life of this patent
Estimating the term…
Abstract
A method, computer program product, and computing system for executing a plurality of requests to process data using a trained machine learning model. An anomalous pattern of requests including at least a threshold amount of out-of-domain data is identified from the plurality of requests. A potential model inversion attack is detected based upon, at least in part, identifying the anomalous pattern of requests.
Claims20
A computing system comprising:
- a memory storing programming instructions;
- and
- a processor configured to execute the programming instructions stored by the memory, wherein the programming instructions, upon execution by the processor, cause the computing system to:
- receive a plurality of requests to process data using a trained machine learning model;
- generate a model inversion confidence score based on the plurality of requests;
- determine that the model inversion confidence score exceeds a first threshold;
- process the plurality of requests using a decoy machine learning model in response to determining that the model inversion confidence score exceeds the first threshold;
- and
- detect a potential model inversion attack on the trained machine learning model by identifying convergence in the plurality of requests and training data associated with the decoy machine learning model.
A computer program product residing on a non-transitory computer readable medium having a plurality of instructions stored thereon which, when executed by a processor, cause the processor to perform operations comprising:
- receiving a plurality of requests to process data using a trained machine learning model;
- identifying, within the plurality of requests, an anomalous pattern of requests;
- generating a model inversion confidence score by comparing the anomalous pattern of requests to training data associated with the trained machine learning model;
- determining that the model inversion confidence score exceeds a first threshold;
- processing the plurality of requests using a decoy machine learning model in response to determining that the model inversion confidence score exceeds the first threshold;
- and
- detecting a potential model inversion attack by identifying convergence in the anomalous pattern of requests and the training data.
A method comprising:
- receiving a plurality of requests to process data using a trained machine learning model;
- identifying, within the plurality of requests, an anomalous pattern of requests;
- generating a model inversion confidence score by comparing the anomalous pattern of requests to training data associated with the trained machine learning model;
- determining that the model inversion confidence score exceeds a first threshold;
- processing the plurality of requests using a decoy machine learning model in response to determining that the model inversion confidence score exceeds the first threshold;
- and
- detecting a potential model inversion attack by identifying convergence in the anomalous pattern of requests and the training data.
Dependencies are inferred from the claim text.