Back to results
US 7,689,582 B2Granted
Data flow system and method for heterogeneous data integration environments
AssigneeInternational Business Machines Corporation
InventorsMarion Else Behnen, Qi Jin, Hui Liao, Sriram K. Padmanabhan, Dinkar Venkatraman Rao, Sriram Srinivasan
Life of this patent
Estimating the term…
Abstract
A computer implemented method, apparatus, and computer usable program code for generating an execution plan graph from a data flow. A metadata representation of the data flow is generated in response to receiving the data flow. A set of code units is generated from the metadata representation. Each code unit in the set of code units is executable on multiple different types of runtime engines. The set of code units is processed to produce the execution plan graph.
Claims30
A computer implemented method for generating an execution plan graph from a data flow, the computer implemented method comprising:
- configuring one or more computer processors to perform an operation comprising:
- receiving the data flow, wherein the data flow includes operators and execution dependencies between the operators, each execution dependency specifying:
- (i) a first operator;
- (ii) and a second operator;
- and (iii) that the second operator is to execute only after the first operator has finished executing;
- and wherein the operators are supported by a plurality of runtime engine types;
- generating, by operation of the one or more computer processors and in response to receiving the data flow, a metadata representation of the data flow to produce a logical operator graph, wherein generating the metadata representation comprises:
- classifying the operators into groups according to the execution dependencies and the plurality of runtime engine types, wherein at least one of the groups comprises a sequence of operators in the data flow that is supported by a single runtime engine type, and wherein the operators are classified to preclude any circular execution dependency among the groups;
- determining, based on the groups and the execution dependencies, an order in which to execute the groups;
- and
- optimizing at least one of the groups based on the runtime engine type of the at least one of the groups;
- generating a set of code units from the metadata representation, wherein each code unit in the set of code units is executable on the plurality of runtime engine types;
- and
- processing the set of code units to produce the execution plan graph.
11 more elements
A computer readable storage medium containing a program which, when executed, performs an operation for generating an execution plan graph from a data flow, the operation comprising:
- receiving the data flow, wherein the data flow includes operators and execution dependencies between the operators, each execution dependency specifying:
- (i) a first operator;
- (ii) and a second operator;
- and (iii) that the second operator is to execute only after the first operator has finished executing;
- and wherein the operators are supported by a plurality of runtime engine types;
- generating, in response to receiving the data flow, a metadata representation of the data flow to produce a logical operator graph, wherein generating the metadata representation comprises:
- classifying the operators into groups according to the execution dependencies and the plurality of runtime engine types, wherein at least one of the groups comprises a sequence of operators in the data flow that is supported by a single runtime engine type, and wherein the operators are classified to preclude any circular execution dependency among the groups;
- determining, based on the groups and the execution dependencies, an order in which to execute the groups;
- and
- optimizing at least one of the groups based on the runtime engine type of the at least one of the groups;
- generating a set of code units from the metadata representation, wherein each code unit in the set of code units is executable on the plurality of runtime engine types;
- and
- processing the set of code units to produce the execution plan graph.
10 more elements
A system, comprising:
- one or more computer processors;
- and
- a memory containing a program, which when executed by the one or more computer processors is configured to perform an operation for generating an execution plan graph from a data flow, the operation comprising:
- receiving the data flow, wherein the data flow includes operators and execution dependencies between the operators, each execution dependency specifying:
- (i) a first operator;
- (ii) and a second operator;
- and (iii) that the second operator is to execute only after the first operator has finished executing;
- and wherein the operators are supported by a plurality of runtime engine types;
- generating, by operation of the one or more computer processors and in response to receiving the data flow, a metadata representation of the data flow to produce a logical operator graph, wherein generating the metadata representation comprises:
- classifying the operators into groups according to the execution dependencies and the plurality of runtime engine types, wherein at least one of the groups comprises a sequence of operators in the data flow that is supported by a single runtime engine type, and wherein the operators are classified to preclude any circular execution dependency among the groups;
- determining, based on the groups and the execution dependencies, an order in which to execute the groups;
- and
- optimizing at least one of the groups based on the runtime engine type of the at least one of the groups;
- generating a set of code units from the metadata representation, wherein each code unit in the set of code units is executable on the plurality of runtime engine types;
- and
- processing the set of code units to produce the execution plan graph.
13 more elements
Dependencies are inferred from the claim text.