# US 2024/0195842 A1 · Massively scalable computer platform using a distributed computational graph

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0195842 A1
- Document ID: US20240195842A1
- Status: Published application
- Kind: A1
- Application number: 18/581,375
- Filed: 2024-02-20
- Published: 2024-06-13
- Earliest claimed: 2015-10-28
- Assignees: Qomplx LLC
- Applicants: Qomplx LLC
- Inventors: Jason Crabtree; Andrew Sellers
- Web page: https://patentgrep.com/patent/US20240195842A1

## CPC

- G06Q 10/063114: {Status monitoring or status determination for a person or group}
- H04L 63/20: {for managing network security; network security policies in general (filtering policies H04L63/0227)}
- G06F 9/5038: {considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration (scheduling strategies G06F9/4881 and subgroups)}
- G06F 16/2477: {Temporal data queries}
- G06F 16/951: Indexing; Web crawling techniques
- G06Q 10/101: Collaborative creation, e.g. joint development of products or services
- G06Q 10/103: {Workflow collaboration or project management}
- H04L 63/1425: {Traffic logging, e.g. anomaly detection}
- H04L 63/1441: {Countermeasures against malicious traffic (countermeasures against attacks on cryptographic mechanisms H04L9/002)}
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/00: Machine learning
- H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/125: involving control of end-device applications over a network
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph using a domain specific language for pipeline and resource declaration has been developed. Data receipt software receives data from one or more sources. In a declared data pathway, data formalization software formats input data for storage, enrichment, or continued processing. An event analysis server inspects stored data for trends, situations, or knowledge. Aggregated data is passed to event message store and data stream handler software. Transformation pipeline software stores, triggers, executes and monitors declared transformation steps and pipelines to manipulate and act on bulk data and data streams, provides results and enables dynamic model and routine re-training and learning.

## Claims

### Claim 1

A computing system for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph employing a massively scalable compute platform, the computing system comprising:
one or more hardware processor configured for:
receiving streams of input from one or more of a plurality of data sources;
filtering data from the stream based on data validity rules;
instantiating a distributed computation graph comprising one or more data transformation pipelines, wherein each transformation pipeline comprises a plurality of nodes which are programmed functions that act on input data and produce output data and a plurality of edges between nodes which are messages carrying data between nodes;
processing the stream using the distributed computation graph; and
during processing of the stream using the distributed computation graph:
analyzing output data from a plurality of nodes of one or more of the plurality of transformation pipelines;
analyzing operational performance of the distributed computation graph;
reconfiguring one or more transformation pipelines based on the analyses of output data and operational performance; and
sending one or more output streams to requesting devices on the network.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein the hardware processor is further configured for:
formatting the data within data stream based upon a set of predetermined parameters so as to prepare for storage in an input event data store; and
performing analysis of data from the input event data store, the data comprising at least one of: a set of trends, a previous occurrence of a progression of a set of events, or an occurrence of a single specific event.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein multiple antecedent node outputs are used as input to one node.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein output from one node acts as input to one or more downstream nodes.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein a transformation pipeline comprises a cyclical subgraph configuration.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the structure of a transformation pipeline is a directed graph.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein at least one node within a transformation pipeline operates as a data store and forms a queue for subsequent nodes in order to serialize the effects of transformation functions at branches or at the head of a cyclical configuration.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein the analysis of the operational performance of the distributed computation graph is based on collected administrative execution and operational data associated with the distributed computational graph; and
wherein the collected administrative execution and operational data comprises at least one of: execution data, failure data, trigger data, and timing data.

### Claim 9 (depends on 8)

The computing system of claim 8, wherein the collected administrative execution and operational data further comprises at least one of: resource usage and input or output data.

### Claim 10

A computer-implemented method executed on a massively scalable compute platform for using a distributed computation graph for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph, the computer-implemented method comprising:
receiving streams of input from one or more of a plurality of data sources;
filtering data from the stream based on data validity rules;
instantiating a distributed computation graph comprising one or more data transformation pipelines, wherein each transformation pipeline comprises a plurality of nodes which are programmed functions that act on input data and produce output data and a plurality of edges between nodes which are messages carrying data between nodes;
processing the stream using the distributed computation graph; and
during processing of the stream using the distributed computation graph:
analyzing output data from a plurality of nodes of one or more of the plurality of transformation pipelines;
analyzing operational performance of the distributed computation graph;
reconfiguring one or more transformation pipelines based on the analyses of output data and operational performance; and
sending one or more output streams to requesting devices on the network.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising the steps of:
formatting the data within data stream based upon a set of predetermined parameters so as to prepare for storage in an input event data store; and
performing batch analysis of data from the input event data store, the data comprising at least one of: a set of trends, a previous occurrence of a progression of a set of events, or an occurrence of a single specific event.

### Claim 12 (depends on 10)

The computer-implemented method of claim 10, wherein multiple antecedent node outputs are used as input to one node.

### Claim 13 (depends on 10)

The computer-implemented method of claim 10, wherein output from one node acts as input to multiple downstream nodes.

### Claim 14 (depends on 10)

The computer-implemented method of claim 10, wherein a transformation pipeline comprises a cyclical subgraph configuration.

### Claim 15 (depends on 10)

The computer-implemented method of claim 10, wherein the structure of a transformation pipeline is a directed graph.

### Claim 16 (depends on 10)

The computer-implemented method of claim 10, wherein at least one node within a transformation pipeline operates as a data store and forms a queue for subsequent nodes in order to serialize the effects of transformation functions at branches or at the head of a cyclical configuration.

### Claim 17 (depends on 10)

The computer-implemented method of claim 10, wherein the analysis of the operational performance of the distributed computation graph is based on collected administrative execution and operational data associated with the distributed computational graph;
wherein the collected administrative execution and operational data comprises at least one of: execution data, failure data, trigger data, and timing data.

### Claim 18 (depends on 17)

The computer-implemented method of claim 17, wherein the collected administrative execution and operational data further comprises at least one of: resource usage and input or output data.

### Claim 19

A system for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph employing a massively scalable compute platform, comprising one or more computers with executable instructions that, when executed, cause the system to:
receive streams of input from one or more of a plurality of data sources;
filter data from the stream based on data validity rules;
instantiate a distributed computation graph comprising one or more data transformation pipelines, wherein each transformation pipeline comprises a plurality of nodes which are programmed functions that act on input data and produce output data and a plurality of edges between nodes which are messages carrying data between nodes;
process the stream using the distributed computation graph; and
during processing of the stream using the distributed computation graph:
analyze output data from a plurality of nodes of one or more of the plurality of transformation pipelines;
analyze operational performance of the distributed computation graph;
reconfigure one or more transformation pipelines based on the analyses of output data and operational performance; and
send one or more output streams to requesting devices on the network.

### Claim 20 (depends on 19)

The system of claim 19, wherein the system is further caused to:
format the data within data stream based upon a set of predetermined parameters so as to prepare for storage in an input event data store; and
perform analysis of data from the input event data store, the data comprising at least one of: a set of trends, a previous occurrence of a progression of a set of events, or an occurrence of a single specific event.

### Claim 21 (depends on 19)

The system of claim 19, wherein multiple antecedent node outputs are used as input to one node.

### Claim 22 (depends on 19)

The system of claim 19, wherein output from one node acts as input to one or more downstream nodes.

### Claim 23 (depends on 19)

The system of claim 19, wherein a transformation pipeline comprises a cyclical subgraph configuration.

### Claim 24 (depends on 19)

The system of claim 19, wherein the structure of a transformation pipeline is a directed graph.

### Claim 25 (depends on 19)

The system of claim 19, wherein at least one node within a transformation pipeline operates as a data store and forms a queue for subsequent nodes in order to serialize the effects of transformation functions at branches or at the head of a cyclical configuration.

### Claim 26 (depends on 19)

The system of claim 19, wherein the analysis of the operational performance of the distributed computation graph is based on collected administrative execution and operational data associated with the distributed computational graph;
wherein the collected administrative execution and operational data comprises at least one of: execution data, failure data, trigger data, and timing data.

### Claim 27 (depends on 26)

The system of claim 26, wherein the collected administrative execution and operational data further comprises at least one of: resource usage and input or output data.

### Claim 28

Non-transitory, computer-readable storage media having computer executable instructions embodied thereon that, when executed by one or more processors of a computing system employing a massively scalable compute platform for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph, cause the computing system to:
receive streams of input from one or more of a plurality of data sources;
filter data from the stream based on data validity rules;
instantiate a distributed computation graph comprising one or more data transformation pipelines, wherein each transformation pipeline comprises a plurality of nodes which are programmed functions that act on input data and produce output data and a plurality of edges between nodes which are messages carrying data between nodes;
process the stream using the distributed computation graph; and
during processing of the stream using the distributed computation graph:
analyze output data from a plurality of nodes of one or more of the plurality of transformation pipelines;
analyze operational performance of the distributed computation graph;
reconfigure one or more transformation pipelines based on the analyses of output data and operational performance; and
send one or more output streams to requesting devices on the network.

### Claim 29 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein the computing system is further caused to:
format the data within data stream based upon a set of predetermined parameters so as to prepare for storage in an input event data store; and
perform analysis of data from the input event data store, the data comprising at least one of: a set of trends, a previous occurrence of a progression of a set of events, or an occurrence of a single specific event.

### Claim 30 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein multiple antecedent node outputs are used as input to one node.

### Claim 31 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein output from one node acts as input to one or more downstream nodes.

### Claim 32 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein a transformation pipeline comprises a cyclical subgraph configuration.

### Claim 33 (depends on 28)

The non-transitory, computer-readable storage media of claim 28 wherein the structure of a transformation pipeline is a directed graph.

### Claim 34 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein at least one node within a transformation pipeline operates as a data store and forms a queue for subsequent nodes in order to serialize the effects of transformation functions at branches or at the head of a cyclical configuration.

### Claim 35 (depends on 28)

The non-transitory, computer-readable storage media of claim 28, wherein the analysis of the operational performance of the distributed computation graph is based on collected administrative execution and operational data associated with the distributed computational graph;
wherein the collected administrative execution and operational data comprises at least one of: execution data, failure data, trigger data, and timing data.

### Claim 36 (depends on 35)

The non-transitory, computer-readable storage media of claim 35, wherein the collected administrative execution and operational data further comprises at least one of: resource usage and input or output data.
