# US 11,115,505 B2 · Facilitating custom content extraction rule configuration for remote capture agents > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,505 B2 - Document ID: US11115505B2 - Status: Granted - Kind: B2 - Application number: 16/404,644 - Filed: 2019-05-06 - Published: 2019-08-29 - Granted: 2021-09-07 - Earliest claimed: 2015-01-29 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Fang I. Hsiao; Clayton S. Ching; Michael R. Dickey; Vladimir A. Shcherbakov; Clint Sharp - Web page: https://patentgrep.com/patent/US11115505B2 ## CPC - H04L 69/22: Parsing or analysis of headers - H04L 43/028: by filtering - H04L 43/0876: Network utilisation, e.g. volume of load or congestion level - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The disclosed embodiments provide a system for extracting custom content from network packets. During operation, the system receives a stream of packets. The system then parses packets in the stream to determine a protocol for each packet. Next, the system applies a custom-content-extraction rule to each packet associated with a target protocol to obtain the extracted content. Then, the system stores the extracted content in events in a data store to facilitate subsequent queries involving the extracted content. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, via a graphical user interface (GUI), input defining a custom content extraction rule, wherein the input specifies: a source field in network packets to be monitored by a remote capture agent, wherein the source field contains structured data, an extraction rule to be used to extract data from the structured data to obtain extracted data, and a field name to be used to identify the extracted data in timestamped events to be generated by the remote capture agent; generating configuration information based on the input; sending the configuration information to the remote capture agent, wherein the configuration information causes the remote capture agent to generate timestamped events, wherein the timestamped events include extracted data obtained by applying the custom content extraction rule to network packets monitored by the remote capture agent, and wherein the extracted data is identified in the timestamped events using the field name; receiving the timestamped events from the remote capture agent, wherein each of the timestamped events includes extracted data identified by the field name; and storing the timestamped events in a data store, wherein storage of the timestamped events in the data store enables execution of queries based on the field name. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the remote capture agent generates a timestamped event of the timestamped events at least in part by: parsing a network packet of the network packets to identify a structure of the network packet, wherein the structure of the network packet is used to determine a protocol associated with the network packets; applying the extraction rule to the network packet to obtain extracted content, wherein applying the extraction rule includes: identifying the source field in the network packet containing the structured data from which the extracted content is to be obtained, and extracting data from the structured data contained in the source field of the network packet; generating a timestamped event including a field storing the extracted content; and sending the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating querying of timestamped event data stored in the data store using late-binding schemas generated from received queries. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the method further comprises: storing the timestamped events in a data store; receiving a query to be applied to the timestamped events stored in the data store; retrieving timestamped events from the data store satisfying the query; using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and processing the query using the retrieved data values. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the input further specifies a protocol to be associated with the custom content extraction rule. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the input further specifies a field-specific regular expression to be applied to the source field in the network packets, and wherein applying the custom content extraction rule to the network packets includes applying the field-specific regular expression to the source field in the network packets. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the structured data includes eXtensible Markup Language (XML) formatted data, and wherein applying the custom content extraction rule includes extracting data from the XML-formatted data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the structured data includes JavaScript Object Notation (JSON) formatted data, and wherein applying the custom content extraction rule includes extracting data from the JSON-formatted data. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the custom content extraction rule is associated with a protocol, and wherein the remote capture agent uses a deep-packet inspection engine to determine that the network packets are associated with the protocol. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the input further specifies an extraction rule type that identifies a type of extraction rule to be used to obtain the extracted data. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the configuration information causes the remote capture agent to send the timestamped events to another component for storage in a data store. ### Claim 11 A non-transitory computer-readable storage medium storing instructions which, when executed by one or more processors, cause performance of operations comprising: receiving, via a graphical user interface (GUI), input defining a custom content extraction rule, wherein the input specifies: a source field in network packets to be monitored by a remote capture agent, the source field containing structured data, second input specifying an extraction rule to be used to extract data from the structured data to obtain extracted data, and a field name to be used to identify the extracted data in timestamped events to be generated by the remote capture agent; generating configuration information based on the input; sending the configuration information to the remote capture agent, wherein the configuration information causes the remote capture agent to generate timestamped events, wherein the timestamped events include extracted data obtained by applying the custom content extraction rule to network packets monitored by the remote capture agent, and wherein the extracted data is identified in the timestamped events using the field name receiving the timestamped events from the remote capture agent, wherein each of the timestamped events includes extracted data identified by the field name; and storing the timestamped events in a data store, wherein storage of the timestamped events in the data store enables execution of queries based on the field name. ### Claim 12 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the remote capture agent generates the timestamped events at least in part by: parsing a network packet of the network packets to identify a structure of the network packet, wherein the structure of the network packet is used to determine a protocol associated with the network packet; applying the extraction rule to the network packet to obtain extracted content, wherein applying the extraction rule includes: identifying the source field in the network packet containing the structured data from which the extracted content is to be obtained, and extracting data from the structured data contained in the source field of the network packet; generating a timestamped event including a field storing the extracted content; and sending the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating querying of timestamped event data stored in the data store using late-binding schemas generated from received queries. ### Claim 13 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the instructions, when executed by the one or more processors, further cause performance of operations comprising: storing the timestamped events in a data store; receiving a query to be applied to the timestamped events stored in the data store; retrieving timestamped events from the data store satisfying the query; using a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and processing the query using the retrieved data values. ### Claim 14 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the input further specifies a protocol to be associated with the custom content extraction rule. ### Claim 15 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the input further specifies a field-specific regular expression to be applied to the source field in the network packets, and wherein applying the custom content extraction rule to the network packets includes applying the field-specific regular expression to the source field in the network packets. ### Claim 16 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the structured data includes eXtensible Markup Language (XML) formatted data, and wherein applying the custom content extraction rule includes extracting data from the XML-formatted data. ### Claim 17 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the structured data includes JavaScript Object Notation (JSON) formatted data, and wherein applying the custom content extraction rule includes extracting data from the JSON-formatted data. ### Claim 18 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the custom content extraction rule is associated with a protocol, and wherein the remote capture agent uses a deep-packet inspection engine to determine that the network packets are associated with the protocol. ### Claim 19 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the input further specifies an extraction rule type that identifies a type of extraction rule to be used to obtain the extracted data. ### Claim 20 (depends on 11) The non-transitory computer-readable storage medium of claim 11, wherein the configuration information causes the remote capture agent to send the timestamped events to another component for storage in a data store. ### Claim 21 An apparatus, comprising: one or more processors; a non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, causes the apparatus to: receive, via a graphical user interface (GUI), input defining a custom content extraction rule, wherein the input specifies: a source field in network packets to be monitored by a remote capture agent, the source field containing structured data, an extraction rule to be used to extract data from the structured data to obtain extracted data, and a field name to be used to identify the extracted data in timestamped events to be generated by the remote capture agent; generate configuration information based on the input; send the configuration information to the remote capture agent, wherein the configuration information causes the remote capture agent to generate timestamped events, wherein the timestamped events include extracted data obtained by applying the custom content extraction rule to network packets monitored by the remote capture agent, and wherein the extracted data is identified in the timestamped events using the field name receive the timestamped events from the remote capture agent, wherein each of the timestamped events includes extracted data identified by the field name; and store the timestamped events in a data store, wherein storage of the timestamped events in the data store enables execution of queries based on the field name. ### Claim 22 (depends on 21) The apparatus of claim 21, wherein the remote capture agent generates the timestamped events at least in part by: parsing a network packet of the network packets to identify a structure of the network packet, wherein the structure of the network packet is used to determine a protocol associated with the network packets; applying the extraction rule to the network packet to obtain extracted content, wherein applying the extraction rule includes: identifying the source field in the network packet containing the structured data from which the extracted content is to be obtained, and extracting data from the structured data contained in the source field of the network packet; generating a timestamped event including a field storing the extracted content; and sending the timestamped event including the extracted content to another component on a computer network for storage in a data store, the data store facilitating querying of timestamped event data stored in the data store using late-binding schemas generated from received queries. ### Claim 23 (depends on 21) The apparatus of claim 21, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: store the timestamped events in a data store; receive a query to be applied to the timestamped events stored in the data store; retrieve timestamped events from the data store satisfying the query; use a late-binding schema generated from the query to retrieve data values from the retrieved timestamped events; and process the query using the retrieved data values. ### Claim 24 (depends on 21) The apparatus of claim 21, wherein the input further specifies a protocol to be associated with the custom content extraction rule. ### Claim 25 (depends on 21) The apparatus of claim 21, wherein the input further specifies a field-specific regular expression to be applied to the source field in the network packets, and wherein applying the custom content extraction rule to the network packets includes applying the field-specific regular expression to the source field in the network packets. ### Claim 26 (depends on 21) The apparatus of claim 21, wherein the structured data includes eXtensible Markup Language (XML) formatted data, and wherein applying the custom content extraction rule includes extracting data from the XML-formatted data. ### Claim 27 (depends on 21) The apparatus of claim 21, wherein the structured data includes JavaScript Object Notation (JSON) formatted data, and wherein applying the custom content extraction rule includes extracting data from the JSON-formatted data. ### Claim 28 (depends on 21) The apparatus of claim 21, wherein the custom content extraction rule is associated with a protocol, and wherein the remote capture agent uses a deep-packet inspection engine to determine that the network packets are associated with the protocol. ### Claim 29 (depends on 21) The apparatus of claim 21, wherein the input further specifies an extraction rule type that identifies a type of extraction rule to be used to obtain the extracted data. ### Claim 30 (depends on 21) The apparatus of claim 21, wherein the configuration information causes the remote capture agent to send the timestamped events to another component for storage in a data store.