Cost-aware replication of intermediate data in dataflows
AssigneeInternational Business Machines Corporation
InventorsClaris Castillo, Malgorzata Steinder, Asser Nasreldin Tantawi
Life of this patent
Estimating the term…
Abstract
Described herein are methods, systems, apparatuses and products for cost-aware replication of intermediate data in dataflows. An aspect provides receiving at least one measurement indicative of a reliability cost associated with executing a dataflow; computing a degree of replication of at least one intermediate data set in the dataflow based on the reliability cost; and communicating at least one replication factor to at least one component of a system responsible for replication of the at least one intermediate data set in the dataflow; wherein the at least one intermediate data set is replicated according to the replication factor. Other embodiments are disclosed.
Claims22
A computer program product comprising:
- a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
- computer readable program code configured to receive at least one measurement indicative of a reliability cost associated with executing a dataflow;
- computer readable program code configured to compute a degree of replication of at least one intermediate data set in the dataflow based on the reliability cost;
- wherein the reliability cost comprises a metric based at least on a sum of two sub-costs, the two sub-costs comprising a cost of replication for the at least one intermediate data set in the dataflow and a cost of regeneration for the at least one intermediate data set in the dataflow;
- computer readable program code configured to minimize the reliability cost;
- and
- computer readable program code configured to communicate at least one replication factor, corresponding to the computed degree of replication, to at least one component responsible for replication of the at least one intermediate data set in the dataflow;
- wherein the at least one intermediate data set is replicated according to the replication factor.
A method comprising:
- receiving at least one measurement indicative of a reliability cost associated with executing a dataflow;
- computing a degree of replication of at least one intermediate data set in the dataflow based on the reliability cost;
- wherein the reliability cost comprises a metric based at least on a sum of two sub-costs, the two sub-costs comprising a cost of replication for the at least one intermediate data set in the dataflow and a cost of regeneration for the at least one intermediate data set in the dataflow;
- minimizing the reliability cost;
- and
- communicating at least one replication factor, corresponding to the computed degree of replication, to at least one component responsible for replication of the at least one intermediate data set in the dataflow;
- wherein the at least one intermediate data set is replicated according to the replication factor.
A system comprising:
- at least one processor;
- and
- a memory device operatively connected to the at least one processor;
- wherein, responsive to execution of program instructions accessible to the at least one processor, the at least one processor is configured to:
- receive at least one measurement indicative of a reliability cost associated with executing a dataflow;
- compute a degree of replication of at least one intermediate data set in the dataflow based on the reliability cost;
- wherein the reliability cost comprises a metric based at least on a sum of two sub-costs, the two sub-costs comprising a cost of replication for the at least one intermediate data set in the dataflow and a cost of regeneration for the at least one intermediate data set in the dataflow;
- minimize the reliability cost;
- and
- communicate at least one replication factor, corresponding to the computed degree of replication, to at least one component responsible for replication of the at least one intermediate data set in the dataflow;
- wherein the at least one intermediate data set is replicated according to the replication factor.
Dependencies are inferred from the claim text.