Method and system for compression and optimization of in-line and in-transit information security data

A system for determining redundant discrete raw alerts from unstructured data-in-transit comprising:

• circuitry configured to generate a movable variable sized time window and to use the movable variable sized time window to encompass a first set of discrete raw alerts in the unstructured data-in-transit, wherein the data-in-transit are arranged in a chronological order, and wherein each discrete raw alert comprises a data component and a time component;
• circuitry configured to generate a digital signature for each discrete raw alert encompassed within the movable variable sized time window using the data component contained in each discrete raw alert;
• circuitry configured to process the generated digital signatures comprising:
• circuitry configured to match a generated digital signature with digital signatures stored in a meta-definition database, if the generated digital signature matches with digital signatures stored in the meta-definition database, the circuitry is configured to add the matched digital signature to a severity threshold database, and if the generated digital signature does not match digital signatures stored in the meta-definition database, the circuitry is configured to add unmatched digital signatures to the meta-definition database,
• whereby, for digital signatures in the severity threshold database that have incident counts exceeding a predetermined incident count threshold, the circuitry is configured to mark such digital signatures as redundant digital signatures and the circuitry is configured to mark raw alerts associated with redundant digital signatures as redundant raw alerts;
• and
• circuitry configured to mark raw alerts having unmatched digital signatures as non-redundant raw alerts, and to aggregate, compress and encrypt non-redundant raw alerts.