Back to results
US 11,080,067 B2Granted
Dynamic execution of parameterized applications for the processing of keyed network data streams
AssigneeAb Initio Technology LLC
InventorsOded Ravid, Trevor Murphy
Life of this patent
Estimating the term…
Abstract
A method is described for processing keyed data items that are each associated with a value of a key, the keyed data items being from a plurality of distinct data streams, the processing including collecting the keyed data items, determining, based on contents of at least one of the keyed data items, satisfaction of one or more specified conditions for execution of one or more actions and causing execution of at least one of the one or more actions responsive to the determining.
Claims40
A data processing system for detecting aggregated values for keyed data items, with each keyed data item associating a particular data item with a particular key value, and for each of one or more key values, causing execution of one or more actions, including:
- a detection sub-system configured to receive a queue of keyed data items collected from a plurality of distinct data sources that are each external to the data processing system, with each keyed data item including a data item with a particular data value for an associated key value, with at least a first keyed data item collected from a first one of the distinct data sources having a data format that is distinct from at least a second keyed data item collected from a second one of the distinct data sources, and with a keyed data item in the queue being reformatted from a format of a data source from which that keyed data item is collected to a readable format of the detection sub-system;
- wherein the detection sub-system includes an in-memory data store for storing, for each of a plurality of key values, one or more data records with a plurality of pre-defined fields each storing one or more aggregated values for that key value, with an aggregated value for that key value being an aggregation of (i) one or more data values for that key value retrieved from another data store, and (ii) one or more data values of one or more respective data items for that key value included in the queue of keyed data items collected from the plurality of distinct data sources;
- wherein the detection sub-system is configured to execute an application for processing keyed data items input into the detection sub-system from the queue, by:
- for a particular key value of an input, keyed data item,
- identify a current state of the application for that particular key value;
- access, from the in-memory data store, a data record for that particular key value;
- execute one or more portions of the application designated for execution in that current state against the accessed data record;
- upon detecting a specified aggregated value in a specified field of the accessed data record, output instructions specifying detection of the specified aggregated value for particular key value;
- cause the application, with respect to the particular key value, to transition from its current state to a subsequent state;
- and
- generate one or more data items for one or more indicators, where one or more values of such one or more data items are based on the particular keyed value and based on execution of the application;
- and
- an instruction sub-system that instructs one or more communication channels to perform one or more actions for the particular key value, responsive to receipt of the instructions for the particular key value from the detection sub-system.
10 more elements
A method implemented by a data processing system for detecting aggregated values for keyed data items, with each keyed data item associating a particular data item with a particular key value, and for each of one or more key values, causing execution of one or more actions, including:
- receiving, by a detection sub-system, a queue of keyed data items collected from a plurality of distinct data sources that are each external to the data processing system, with each keyed data item including a data item with a particular data value for an associated key value, with at least a first keyed data item collected from a first one of the distinct data sources having a data format that is distinct from at least a second keyed data item collected from a second one of the distinct data sources, and with a keyed data item in the queue being reformatted from a format of a data source from which that keyed data item is collected to a readable format of the detection sub-system;
- storing, in in-memory data store for each of a plurality of key values, one or more data records with a plurality of pre-defined fields each storing one or more aggregated values for that key value, with an aggregated value for that key value being an aggregation of (i) one or more data values for that key value retrieved from another data store, and (ii) one or more data values of one or more respective data items for that key value included in the queue of keyed data items collected from the plurality of distinct data sources;
- executing an application for processing keyed data items input into the detection sub-system from the queue, by:
- for a particular key value of an input, keyed data item,
- identifying a current state of the application for that particular key value;
- accessing, from the in-memory data store, a data record for that particular key value;
- executing one or more portions of the application designated for execution in that current state against the accessed data record;
- upon detecting a specified aggregated value in a specified field of the accessed data record, outputting instructions specifying detection of the specified aggregated value for particular key value;
- causing the application, with respect to the particular key value, to transition from its current state to a subsequent state;
- and
- generating one or more data items for one or more indicators, where one or more values of such one or more data items are based on the particular keyed value and based on execution of the application;
- and
- instructing one or more communication channels to perform one or more actions for the particular key value, responsive to receipt of the instructions for the particular key value from the detection sub-system.
10 more elements
One or more machine-readable hardware storage devices for detecting aggregated values for keyed data items, with each keyed data item associating a particular data item with a particular key value, and for each of one or more key values, causing execution of one or more actions, the one or more machine-readable hardware storage devices storing instructions that are executable by one or more processing devices to perform operations including:
- receiving a queue of keyed data items collected from a plurality of distinct data sources that are each external to the data processing system, with each keyed data item including a data item with a particular data value for an associated key value, with at least a first keyed data item collected from a first one of the distinct data sources having a data format that is distinct from at least a second keyed data item collected from a second one of the distinct data sources, and with a keyed data item in the queue being reformatted from a format of a data source from which that keyed data item is collected to a readable format;
- storing, in in-memory data store for each of a plurality of key values, one or more data records with a plurality of pre-defined fields each storing one or more aggregated values for that key value, with an aggregated value for that key value being an aggregation of (i) one or more data values for that key value retrieved from another data store, and (ii) one or more data values of one or more respective data items for that key value included in the queue of keyed data items collected from the plurality of distinct data sources;
- executing an application for processing keyed data items input into a detection sub-system from the queue, by:
- for a particular key value of an input, keyed data item,
- identifying a current state of the application for that particular key value;
- accessing, from the in-memory data store, a data record for that particular key value;
- executing one or more portions of the application designated for execution in that current state against the accessed data record;
- upon detecting a specified aggregated value in a specified field of the accessed data record, outputting instructions specifying detection of the specified aggregated value for particular key value;
- causing the application, with respect to the particular key value, to transition from its current state to a subsequent state;
- and
- generating one or more data items for one or more indicators, where one or more values of such one or more data items are based on the particular keyed value and based on execution of the application;
- and
- instructing one or more communication channels to perform one or more actions for the particular key value, responsive to receipt of the instructions for the particular key value from the detection sub-system.
10 more elements
Dependencies are inferred from the claim text.