# US 10,873,467 B2 · Method and system for compression and optimization of in-line and in-transit information security data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,467 B2 - Document ID: US10873467B2 - Status: Granted - Kind: B2 - Application number: 16/077,363 - Filed: 2016-02-15 - Published: 2019-02-21 - Granted: 2020-12-22 - Earliest claimed: 2016-02-15 - Assignees: Certis Cisco Security Pte Ltd - Applicants: Certis Cisco Security Pte Ltd - Inventors: Keng Leng Albert Lim - Web page: https://patentgrep.com/patent/US10873467B2 ## CPC - H04L 9/3247: {involving digital signatures} - G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)} - G06F 16/313: {Selection or weighting of terms for indexing} - G06F 21/55: Detecting local intrusion or implementing counter-measures - H04L 9/3239: {involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD} - H04L 9/3297: {involving time stamps, e.g. generation of time stamps} - H04L 63/1416: {Event detection, e.g. attack signature detection} - H04L 63/1425: {Traffic logging, e.g. anomaly detection} - G06F 2221/2125: Just-in-time application of countermeasures, e.g., on-the-fly decryption, just-in-time obfuscation or de-obfuscation - H04L 2209/30: Compression, e.g. Merkle-Damgard construction - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract This document discloses a method and system for just-in-time compression and optimization of raw unstructured in-line and in-transit data by identifying low entropy data blocks or duplicated information security information in raw computer security alerts within a series of time windows. In particular, the method and system automatically manages; processes; and optimizes in-line and in-transit data blocks or raw information security alerts received from a plurality of information surveillance sources and/or peripheral monitoring devices simultaneously. The data blocks or raw information security alerts that are found to be unique in the various time windows are transposed into meta-definition tables to be further processed while redundant data blocks or raw alerts contained within each particular time window are identified, marked and processed accordingly. ## Claims ### Claim 1 A method for determining redundant discrete raw alerts from unstructured data-in-transit, the method to be performed by a computer system comprising: generating a movable variable sized time window and using the movable variable sized time window to encompass a first set of discrete raw alerts in the unstructured data-in-transit, wherein the data-in-transit are arranged in a chronological order, and wherein each discrete raw alert comprises a data component and a time component; generating a digital signature for each discrete raw alert encompassed within the variable sized time window using the data component contained in each discrete raw alert; processing the generated digital signatures, the processing of each generated digital signature comprises: matching a generated digital signature with digit al signatures stored in a meta-definition database, if the generated digital signature matches with digital signatures stored in the meta-definition database, adding the matched digital signature to a severity threshold database, and if the generated digital signature does not match digital signatures stored in the meta-definition database, adding unmatched digital signatures to the meta-definition database, whereby, for digital signatures in the severity threshold database that have incident counts exceeding a predetermined incident count threshold, marking such digital signatures as redundant digital signatures and marking raw alerts associated with redundant digital signatures as redundant raw alerts; and marking raw alerts having unmatched digital signatures as non-redundant raw alerts, and aggregating, compressing and encrypting non-redundant raw alerts. ### Claim 2 (depends on 1) The method according to claim 1 further comprising: generating the movable variable sized time window and moving the variable sized time window to encompass a second set of discrete raw alerts in the unstructured data-in-transit, wherein the second set of discrete raw alerts are later in the chronological order than the first set of discrete raw alerts; generating a digital signature for each discrete raw alert encompassed within the variable sized time window using the data component contained in each discrete raw alert; processing the generated digital signatures, the processing of each generated digital signature comprises: matching a generated digital signature with digital signatures stored in a meta-definition database, if the generated digital signature matches with digital signatures stored in the meta-definition database, adding the matched digital signature to a severity threshold database, and if the generated digital signature does not match digital signatures stored in the meta-definition database, adding unmatched digital signatures to the meta-definition database, whereby, for digital signatures in the severity threshold database that have incident counts exceeding a predetermined incident count threshold, marking such digital signatures as redundant digital signatures and making raw alerts associated with redundant digital signature as redundant raw alerts; and making raw alerts having unmatched digital signatures as non-redundant raw alerts, and aggregating, compressing and encrypting non-redundant raw alerts. ### Claim 3 (depends on 1) The method according to claim 1 wherein digital signatures that are marked as redundant digital signatures are further marked with a time stamp indicating when the digital signature was marked as redundant whereby, after a period of time from the time stamp, redundant digital signatures are removed from the severity threshold database. ### Claim 4 (depends on 2) The method according to claim 2 further comprising: removing redundant raw alerts from the unstructured data-in-transit. ### Claim 5 (depends on 1) The method according to claim 1 wherein the generating the digital signature for each discrete raw alert comprises: using a cryptographic hash function to generate the digital signature whereby the digital signature comprises a hash value. ### Claim 6 (depends on 5) The method according to claim 5 wherein the cryptographic hash function comprises a 256-bit Secure Hash Algorithm (SHA). ### Claim 7 (depends on 2) The method according to claim 2 wherein the generating the movable variable sized time window comprises: dynamically changing a size of the movable variable sized time window, wherein the size of the movable variable sized time window is increased if a ratio of non-redundant raw alerts to redundant raw alerts is higher than a predetermined ratio. ### Claim 8 A system for determining redundant discrete raw alerts from unstructured data-in-transit comprising: circuitry configured to generate a movable variable sized time window and to use the movable variable sized time window to encompass a first set of discrete raw alerts in the unstructured data-in-transit, wherein the data-in-transit are arranged in a chronological order, and wherein each discrete raw alert comprises a data component and a time component; circuitry configured to generate a digital signature for each discrete raw alert encompassed within the movable variable sized time window using the data component contained in each discrete raw alert; circuitry configured to process the generated digital signatures comprising: circuitry configured to match a generated digital signature with digital signatures stored in a meta-definition database, if the generated digital signature matches with digital signatures stored in the meta-definition database, the circuitry is configured to add the matched digital signature to a severity threshold database, and if the generated digital signature does not match digital signatures stored in the meta-definition database, the circuitry is configured to add unmatched digital signatures to the meta-definition database, whereby, for digital signatures in the severity threshold database that have incident counts exceeding a predetermined incident count threshold, the circuitry is configured to mark such digital signatures as redundant digital signatures and the circuitry is configured to mark raw alerts associated with redundant digital signatures as redundant raw alerts; and circuitry configured to mark raw alerts having unmatched digital signatures as non-redundant raw alerts, and to aggregate, compress and encrypt non-redundant raw alerts. ### Claim 9 (depends on 8) The system according to claim 8 further comprising: circuitry configured to generate the movable variable sized time window and move the movable variable sized time window to encompass a second set of discrete raw alerts in the unstructured data-in-transit, wherein the second set of discrete raw alerts are later in the chronological order than the first set of discrete raw alerts; circuitry configured to generate a digital signature for each discrete raw alert encompassed within the time window using the data component contained in each discrete raw alert; circuitry configured to process the generated digital signatures comprising: circuitry configured to match a generated digital signature with digital signatures stored in a meta-definition database, if the generated digital signature matches with digital signatures stored in the meta-definition database, the circuitry is configured to add the matched digital signature to a severity threshold database, and if the generated digital signature does not match digital signatures stored in the meta-definition database, the circuitry is configured to add unmatched digital signatures to the meta-definition database, whereby, for digital signatures in the severity threshold database that have incident counts exceeding a predetermined incident count threshold, the circuitry is configured to mark such digital signatures as redundant digital signatures and the circuitry is configured to mark raw alerts associated with redundant digital signatures as redundant raw alerts; and circuitry configured to mark raw alerts having unmatched digital signatures as non-redundant raw alerts, and to aggregate, compress and encrypt non-redundant raw alerts. ### Claim 10 (depends on 8) The system according to claim 8 wherein digital signatures that are marked as redundant digital signatures are further marked with a time stamp indicating when the digital signature was marked as redundant whereby, after a period of time from the time stamp, the circuitry is configured to remove redundant digital signatures from the severity threshold database. ### Claim 11 (depends on 8) The system according to claim 8 further comprising: circuitry configured to remove redundant raw alerts from the unstructured data-in-transit. ### Claim 12 (depends on 8) The system according to claim 8 wherein the circuitry configured to generate the digital signature for each discrete raw alert comprises: circuitry configured to use a cryptographic hash function to generate the digital signature whereby the digital signature comprises a hash value. ### Claim 13 (depends on 12) The system according to claim 12 wherein the cryptographic hash function comprises a 256-bit Secure lash Algorithm (SHA). ### Claim 14 (depends on 8) The system according to claim 8 wherein the circuitry configured to generate the movable variable sized time window comprises: circuitry configured to dynamically change a size of the movable variable sized time window wherein the size of the movable variable sized time window is increased if a ratio of non-redundant raw alerts to redundant raw alerts is higher than a 10:1 ratio.