# US 10,313,377 B2 · Universal link to extract and classify log data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,377 B2 - Document ID: US10313377B2 - Status: Granted - Kind: B2 - Application number: 15/298,150 - Filed: 2016-10-19 - Published: 2018-04-19 - Granted: 2019-06-04 - Earliest claimed: 2016-10-19 - Assignees: Anomali Incorporated - Applicants: Anomali Incorporated - Inventors: Wei Huang; Yizheng Zhou; Hugh Seretse Njemanze; Zhong Deng - Web page: https://patentgrep.com/patent/US10313377B2 ## CPC - H04L 63/1425: {Traffic logging, e.g. anomaly detection} - G06F 16/285: {Clustering or classification} - G06F 16/906: Clustering; Classification - G06F 16/2255: {Hash tables} - G06F 16/245: Query processing - H04L 2463/121: Timestamp - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A universal link to extract and classify log data is disclosed. In various embodiments, a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest is identified. The candidate data values are processed through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific data value type. Candidates, if any, which match the more specific pattern are classified as being of a corresponding specific data type and are removed from the set of candidate data values. A structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest is generated and stored. ## Claims ### Claim 1 A system, comprising: a memory configured to store arbitrary log data; and a processor coupled to the memory and configured to: identify in said arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest; process said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest; classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and removing from the set of candidate data values any candidate data values so identified and classified; and generate and store a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest; wherein the processor is further configured to apply one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest; and wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string; placement within the log data of such a string relative to a given candidate data value; location of a given candidate data value within the arbitrary log data; and location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type. ### Claim 2 (depends on 1) The system of claim 1, wherein said arbitrary log data comprises a discrete set of log data. ### Claim 3 (depends on 2) The system of claim 2, wherein said arbitrary log data comprises a line of log data. ### Claim 4 (depends on 1) The system of claim 1, wherein the processor is configured to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data. ### Claim 5 (depends on 1) The system of claim 1, wherein said arbitrary log data may include log data expressed and represented in any format. ### Claim 6 (depends on 1) The system of claim 1, wherein said arbitrary log data may include zero, one, or more occurrences of each of said types of data value of interest. ### Claim 7 (depends on 1) The system of claim 1, wherein said types of data value of interest include one or more of the following: an IP address type, a domain type, a URL type, and an email address type. ### Claim 8 (depends on 1) The system of claim 1, wherein the processor is further configured to discard candidate values comprising the set of candidate values that do not match any of said more specific patterns. ### Claim 9 (depends on 1) The system of claim 1, wherein the processor is further configured to detect that said arbitrary log data includes more than a prescribed threshold number of occurrences of data values of a given type, and to split the arbitrary log data into two or more sets each including the prescribed number or fewer occurrences of data values of said given type. ### Claim 10 (depends on 9) The system of claim 9, wherein the processor is configured to perform said recited steps to identify, process, and generate and store separately with respect to each set of log data generated by splitting the arbitrary log data into two or more sets. ### Claim 11 (depends on 1) The system of claim 1, further comprising a communication interface configured to receive said arbitrary log data. ### Claim 12 (depends on 1) The system of claim 1, wherein the processor is further configured to check a candidate data value determined to match a more specific pattern associated with a given type against a database of known values of that type. ### Claim 13 (depends on 12) The system of claim 12, wherein the processor is further configured to check said candidate data value determined to match a more specific pattern associated with said given type against said database of known values of that type at least in part using a data structure that has been marked to reflect at least a subset of values in said database. ### Claim 14 (depends on 13) The system of claim 13, wherein said data structure comprises a bloom filter. ### Claim 15 A method, comprising: using a processor to identify in an arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest; using the processor to process said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest; classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and removing from the set of candidate data values any candidate data values so identified and classified; and using the processor to generate and store a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest; wherein the processor is further used to apply one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest; and wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string; placement within the log data of such a string relative to a given candidate data value; location of a given candidate data value within the arbitrary log data; and location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type. ### Claim 16 (depends on 15) The method of claim 15, wherein the processor is configured to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data. ### Claim 17 (depends on 15) The method of claim 15, wherein said arbitrary log data may include log data expressed and represented in any format. ### Claim 18 A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for: identifying in an arbitrary log data a set of candidate data values that match a top level pattern that is common to two or more types of data value of interest; processing said candidate data values through a plurality of successive filtering stages, each stage of which includes determining which, if any, of said candidates match a more specific pattern associated more specifically with a specific one of said types of data value of interest; classifying said candidates, if any, that match the more specific pattern as being of said corresponding specific one of said types of data value of interest; and removing from the set of candidate data values any candidate data values so identified and classified; generating and storing a structured data record that associates each candidate data value determined to be of a corresponding one of said types of data value of interest with said corresponding one of said types of data value of interest; and applying one or more heuristics to more specifically classify and label one or more values determined to match a pattern associated with a specific one of said types of data value of interest; wherein said heuristics include heuristics based on one or more of presence in the arbitrary log data of a characteristic string; placement within the log data of such a string relative to a given candidate data value; location of a given candidate data value within the arbitrary log data; and location within the arbitrary log data of a given candidate data value relative to one or more other candidate data values of the same type. ### Claim 19 (depends on 18) The computer program product of claim 18, further comprising computer instructions to identify said set of candidate data value at least in part by applying a regular expression to text comprising said arbitrary log data. ### Claim 20 (depends on 18) The computer program product of claim 18, wherein said arbitrary log data may include log data expressed and represented in any format.