Security feature extraction for a network

A computer program product comprising a non-transitory computer-readable storage medium configured to store executable computer code that, when executed by a processor, causes the processor to perform the steps of:

• receiving, at a security analytics system, raw data describing a plurality of entities within a local network, the raw data comprising at least system log data and network traffic data, the plurality of entities comprising at least one user and at least one device in the local network;
• parsing the raw data into a plurality of data fields, each data field comprising a type and a value;
• identifying a subset of the plurality of data fields based on the relevance of the plurality of data fields for detecting a security threat in the local network;
• generating filtered data from the raw data, the filtered data comprising the subset of the plurality of data fields;
• generating structured data from the filtered data, the structured data being divided into portions that are each structured in accordance with a schema in a set of schema, each schema describing at least one data field of the subset of data fields from the filtered data;
• identifying relationships between the plurality of entities in the local network based on the structured data, wherein identifying the relationships between the plurality of entities comprises:
• generating an entity graph describing the entity relationships, wherein nodes of the entity graph represent one or more of the plurality of entities, and edges in the graph represent relationships between the entities;
• generating a set of features based on the set of structured data and the identified relationships between the plurality of entities in the local network;
• generating one or more threat scores for one or more entities of the plurality of entities based on the set of features generated from the structured data and the identified relationships, wherein generating one or more threat scores for an entity comprises:
• applying one or more machine-learned models to one or more features of the set of features;
• and
• generating a threat score from each machine-learned model output indicating a likelihood that the entity is performing malicious behavior;
• determining malicious behavior exhibited by a particular entity of the plurality of entities based on the one or more threat scores satisfying a threshold;
• and
• in response to the determining, restricting access by the particular entity to data of the local network.

A system comprising:

• one or more processors and one or more computer-readable storage medium configured to store executable computer code that, when executed by the one or more processors, causes the one or more processors to perform operations comprising:
• receiving, at a security analytics system, raw data describing a plurality of entities within a local network, the raw data comprising at least system log data and network traffic data, the plurality of entities comprising at least one user and at least one device in the local network;
• parsing the raw data into a plurality of data fields, each data field comprising a type and a value;
• identifying a subset of the plurality of data fields based on the relevance of the plurality of data fields for detecting a security threat in the local network;
• generating filtered data from the raw data, the filtered data comprising the subset of the plurality of data fields;
• generating structured data from the filtered data, the structured data being divided into portions that are each structured in accordance with a schema in a set of schema, each schema describing at least one data field of the subset of data fields from the filtered data;
• identifying relationships between the plurality of entities in the local network based on the structured data, wherein identifying the relationships between the plurality of entities comprises:
• generating an entity graph describing the entity relationships, wherein nodes of the entity graph represent one or more of the plurality of entities, and edges in the graph represent relationships between the entities;
• generating a set of features based on the set of structured data and the identified relationships between the plurality of entities in the local network;
• generating one or more threat scores for one or more entities of the plurality of entities based on the set of features generated from the structured data and the identified relationships, wherein generating one or more threat scores for an entity comprises:
• applying one or more machine-learned models to one or more features of the set of features;
• and
• generating a threat score from each machine-learned model output indicating a likelihood that the entity is performing malicious behavior;
• determining malicious behavior exhibited by a particular entity of the plurality of entities based on the one or more threat scores satisfying a threshold;
• and
• in response to the determining, restricting access by the particular entity to data of the local network.