Geo-distributed computation and analytics using an input graph
AssigneeLogRhythm, Inc.
InventorGeoffrey A. Mattson
Life of this patent
Estimating the term…
Abstract
A Multi-site Integrated Session-oriented Transaction (MIST) computing system includes a MIST mapper configured to process a compiled distributed compute session request to determine, for each of at least one remote sets of data, whether to process the remote set of data with one of one or more field nodes or with a cloud node based on a cost of transporting the remote set of data from the proximate field node to the cloud node. In response to determining to process the remote set of data with the cloud node, the MIST mapper is configured to output a data retrieval request to the proximate field node. In response to determining to process the remote set of data with the field node, the MIST mapper is configured to output the computation request to the proximate field node.
Claims19
A system comprising processing circuitry configured to:
- process a computation request for a dataset to generate a compiled request comprising a plurality of functions, wherein the dataset comprises one or more geographically-distributed sets of data, the sets of data including a set of data that is located on or sourced by a device proximate to a proximate field node of a plurality of field nodes;
- determine a cost function based on a cost of transporting the set of data from the proximate field node to one or more other field nodes of the plurality of field nodes using a network deployment of the plurality of field nodes and a computational cost for processing the set of data using the proximate field node;
- process, based on the cost function, an input graph mapping the plurality of functions to the plurality of field nodes to generate an updated graph that modifies the input graph to map a first function of the plurality of functions to the proximate field node and to map a second function of the plurality of functions to a first field node of the plurality of field nodes, wherein to process the input graph, the processing circuitry is further configured to:
- determine a layer of the input graph with a fewest number of nodes, wherein each node in the layer does not depend on any other nodes in the layer;
- and
- move a mapping of the second function in the input graph from a second field node of the plurality of fields nodes to the first field node based on a determination that the second function is in the layer with the fewest number of nodes, wherein the second field node is mapped, in the updated graph, to one or more functions of the plurality of functions of another layer of the input graph that is different from the layer with the fewest number of nodes;
- output, based on the updated graph mapping the first function to the proximate field node, a first request to the proximate field node to cause the proximate field node to process the set of data using the first function and output first modified data;
- and
- output, based on the updated graph mapping the second function to the first field node, a second request to the first field node, wherein the second request is configured to cause the first field node to generate, based on the first modified data and using the second function, second modified data.
A method comprising:
- processing, by processing circuitry, a computation request for a dataset to generate a compiled request comprising a plurality of functions, wherein the dataset comprises one or more geographically-distributed sets of data, the sets of data including a set of data that is located on or sourced by a device proximate to a proximate field node of a plurality of field nodes;
- determining, by the processing circuitry, a cost function based on a cost of transporting the set of data from the proximate field node to one or more other field nodes of the plurality of field nodes using a network deployment of the plurality of field nodes and a computational cost for processing the set of data using the proximate field node;
- processing, by the processing circuitry and based on the cost function, an input graph mapping the plurality of functions to the plurality of field nodes to generate an updated graph that modifies the input graph to map a first function of the plurality of functions to the proximate field node and to map a second function of the plurality of functions to a first field node of the plurality of field nodes, wherein processing the input graph further comprises:
- determining a layer of the input graph with a fewest number of nodes, wherein each node in the layer does not depend on any other nodes in the layer;
- and
- moving a mapping of the second function in the input graph from a second field node of the plurality of fields nodes to the first field node based on a determination that the second function is in the layer with the fewest number of nodes, wherein the second field node is mapped, in the updated graph, to one or more functions of the plurality of functions of another layer of the input graph that is different from the layer with the fewest number of nodes;
- outputting, by the processing circuitry and based on the updated graph mapping the first function to the proximate field node, a first request to the proximate field node to cause the proximate field node to process the set of data using the first function and output first modified data;
- and
- outputting, by the processing circuitry and based on the updated graph mapping the second function to the first field node, a second request to the first field node, wherein the second request is configured to cause the first field node to generate, based on the first modified data and using the second function, second modified data.
Non-transitory computer-readable storage media comprising instructions that, when executed, cause processing circuitry to:
- process a computation request for a dataset to generate a compiled request comprising a plurality of functions, wherein the dataset comprises one or more geographically-distributed sets of data, the sets of data including a set of data that is located on or sourced by a device proximate to a proximate field node of a plurality of field nodes;
- determine a cost function based on a cost of transporting the set of data from the proximate field node to one or more other field nodes of the plurality of field nodes using a network deployment of the plurality of field nodes and a computational cost for processing the set of data using the proximate field node;
- process, based on the cost function, an input graph mapping the plurality of functions to the plurality of field nodes to generate an updated graph that modifies the input graph to map a first function of the plurality of functions to the proximate field node and to map a second function of the plurality of functions to a first field node of the plurality of field nodes, wherein to process the input graph, the instructions further cause the processing circuitry to:
- determine a layer of the input graph with a fewest number of nodes, wherein each node in the layer does not depend on any other nodes in the layer;
- and
- move a mapping of the second function in the input graph from a second field node of the plurality of fields nodes to the first field node based on a determination that the second function is in the layer with the fewest number of nodes, wherein the second field node is mapped, in the updated graph, to one or more functions of the plurality of functions of another layer of the input graph that is different from the layer with the fewest number of nodes;
- output, based on the updated graph mapping the first function to the proximate field node, a first request to the proximate field node to cause the proximate field node to process the set of data using the first function and output first modified data;
- and
- output, based on the updated graph mapping the second function to the first field node, a second request to the first field node, wherein the second request is configured to cause the first field node to generate, based on the first modified data and using the second function, second modified data.
Dependencies are inferred from the claim text.