Machine Learning Based Anomaly Detection

A method of detecting anomalies in an event stream of security-related events of one or more organizations, the method including:

• receiving a security-related event that includes a space identifier (ID) and features with certain values;
• transforming feature-value pairs into categorical bins of a hash-space;
• applying a hash function to a combination of the space ID and the transformed feature-value pairs to retrieve likelihood coefficients for the transformed feature-value pairs and a standard candle for the space ID;
• scoring the likelihood coefficients and any transformed feature-value pairs that do not have likelihood coefficients because the transformed feature-value pairs that do not have likelihood coefficients were not previously observed for the space ID, in combination with evaluating the standard candle, to produce an anomaly score;
• when the anomaly score represents a detected anomaly event, accessing history associated with the space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior security-related events for the space ID;
• and
• invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected.

A non-transitory computer readable storage medium impressed with computer program instructions to detect an anomaly event that has not frequently been observed in an ongoing event stream of security-related events of one or more organizations, which computer program instructions, when executed on a processor, implement a method comprising:

• evaluating a plurality of production events with production space IDs, including for a production event:
• transforming features of the production event into categorical bins of a hash-space;
• applying a hash function to the production space ID and the features of the production event as transformed to retrieve likelihood coefficients for the transformed features of the production event and a standard candle for the production space ID;
• calculating an anomaly score;
• when the anomaly score represents a detected anomaly event, accessing history associated with the production space ID to construct a contrast between feature-event pairs of the detected anomaly event and non-anomalous feature-value pairs of prior events for the production space ID;
• and
• invoking one or more security actions including at least one of a quarantine, and an encryption, to be performed when anomalies are detected;
• wherein the likelihood coefficients had been calculated by space ID and a standard candle and mapped into the hash-space using a loosely supervised machine learning of observed features in security-related events using a loss function analyzer and recording the standard candle.