Optimization for real-time, parallel execution of models for extracting high-value information from data streams

A system for identification of high-value information in data streams, comprising:

• a plurality of processors;
• memory storing one or more programs to be executed by the plurality of processors;
• the one or more programs comprising instructions for:
• receiving a plurality of filter graphs, each filter graph includes a plurality of filter nodes interrelated by a plurality of graph edges, each filter node representing a classification model;
• performing a continuous monitoring process for a data stream that includes a plurality of data packets from a plurality of sources, including:
• without user intervention, in response to receiving the data stream with the plurality of data packets, distributing the plurality of data packets to inputs of the plurality of filter graphs;
• and
• identifying respective ones of the plurality of data packets with high-value information related to a particular concept with regard to the respective filter graphs, based on parallel execution of the filter nodes included in the respective filter graph, by applying predefined criteria associated with the particular concept with respect to classification models of corresponding filter nodes to text content and author information associated with the plurality of data packets,
• wherein applying the predefined criteria to the text content and the author information of a respective data packet of the plurality of data packets includes:
• executing one or more textual filters on the text content of the data packet and one or more author filters on the author information of the data packets in accordance with a first of the classification models;
• and
• determining whether to tag the data packet with an identifier of the first classification model or to tag the data packet as rejected with the identifier of the first classification model based on whether the data packet is accepted by the first classification model;
• upon determining that at least one of the plurality of data packets includes high-value information related to the particular concept, generating statistical information related to the high-value information;
• and
• upon determining that the statistical information meets or exceeds a predetermined threshold, triggering an alarm, including sending a notification to a designated recipient indicating that the predetermined threshold has been met or exceeded.

A method for identification of high-value information in data streams, comprising:

• at a computer system including a plurality of processors and memory storing programs for execution by the processors:
• receiving a plurality of filter graphs, each filter graph including a plurality of filter nodes interrelated by a plurality of graph edges, each filter node representing a classification model;
• performing a continuous monitoring process for a data stream that includes a plurality of data packets from a plurality of sources, including:
• without user intervention, in response to receiving the data stream with the plurality of data packets, distributing the plurality of data packets to inputs of the plurality of filter graphs;
• and
• identifying respective ones of the plurality of data packets with high-value information related to a particular concept with regard to the respective filter graphs, based on parallel execution of the filter nodes included in the respective filter graphs, by applying predefined criteria associated with the particular concept with respect to classification models of corresponding filter nodes to text content and author information associated with the plurality of data packets;
• wherein applying the predefined criteria to the text content and the author information of a respective data packet of the plurality of data packets includes:
• executing one or more textual filters on the text content of the data packet and one or more author filters on the author information of the data packets in accordance with a first of the classification models;
• and
• in accordance with a determination that the data packet is not accepted by the first of the classification models, tagging the data packet as rejected with the identifier of the first of the classification models;
• upon determining that at least one of the plurality of data packets includes high-value information related to the particular concept, generating statistical information related to the high-value information;
• and
• upon determining that the statistical information meets or exceeds a predetermined threshold, triggering an alarm, including sending a notification to a designated recipient indicating that the predetermined threshold has been met or exceeded.