Security model training and threshold selection
AssigneeMicrosoft Technology Licensing, LLC
InventorsPengcheng LUO, Reeves Hoppe BRIGGS, Bryan Robert JEFFREY, Naveed Azeemi AHMAD
Life of this patent
Estimating the term…
Abstract
A set of candidate malicious activity identification models are trained and evaluated against a production malicious activity identification model to identify a best performing model. If the best performing model is one of the candidate models, then an alert threshold is dynamically set for the best performing model, for each of a plurality of different urgency levels. A reset threshold, for each urgency level, is also dynamically set for the best performing model.
Claims20
A computing system, comprising:
- model training logic that automatically trains a set of candidate models to identify malicious activity in a monitored computing system based on activity indicators obtained from the monitored computing system;
- model evaluation logic that automatically evaluates a performance of each of the candidate models and a production model used by an alert generation computing system, based on a set of test data, and that generates a performance metric corresponding to each candidate model and corresponding to the production model;
- and
- model promotion logic that automatically deploys a best performing model, based on the corresponding performance metrics, to the alert generation computing system.
The computing system of claim 5 wherein the dynamic alert threshold setting logic configures the candidate models to set the alert threshold for each urgency level by identifying a confidence score, above which the model generates no more than a first predefined number of false positive outputs that identify input activity test data, that represents benign activity, as representing malicious activity.
A computer implemented method, comprising:
- automatically identifying a set of training data and test data that represent malicious activity and benign activity on a computing system;
- automatically training a set of candidate models, with the training data, to identify malicious activity in a monitored computing system based on activity indicators obtained from the monitored computing system;
- automatically evaluating a performance of each of the candidate models and a production model used by an alert generation computing system, based on the test data, and based on a performance metric corresponding to each candidate model and corresponding to the production model;
- and
- automatically deploying a best performing model, based on the corresponding performance metrics, to the alert generation computing system.
The computer implemented method of claim 14 wherein automatically configuring each candidate model to set a corresponding alert threshold for each urgency level comprises:
- identifying a confidence score, above which the model generates no more than a predefined number of false positive outputs that identify input activity test data, that represents benign activity, as representing malicious activity.
A computing system, comprising:
- model training logic that automatically trains a set of candidate models to identify malicious activity in a monitored computing system based on activity indicators obtained from the monitored computing system, the candidate models and a production model used by an alert generation computing system each being configured to receive input activity data indicative of activities and generate an output indicative of whether the model has determined that the input activity data represents malicious activity and an associated confidence score indicative of a confidence that the model has in the determination that the input activity data represents malicious activity, and to assign the malicious activity to an urgency level by comparing the confidence score associated with the malicious activity to an alert threshold for the urgency level;
- dynamic alert threshold setting logic configured to automatically configure each candidate model to set a corresponding alert threshold for each urgency level;
- model evaluation logic that automatically evaluates a performance of each of the candidate models and the production model, based on a set of test data, and that generates a performance metric indicative of the performance of each candidate model and the production model;
- and
- model promotion logic that automatically deploys a best performing model, based on the corresponding performance metrics, to the alert generation computing system.
Dependencies are inferred from the claim text.