In-Memory Dataflow Execution with Dynamic Placement of Cache Operations

A computer program product, comprising a tangible machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

• obtaining a first cache placement plan for a dataflow comprised of a plurality of operations, wherein said first cache placement plan comprises a combination of output datasets of a subset of said operations in said dataflow to cache based on an estimated reduction in a total execution cost for the dataflow when said first cache placement plan is implemented given an input dataset;
• executing one or more operations of said dataflow and updating a number of references to said one or more executed operations to reflect remaining executions of said one or more executed operations in the dataflow;
• determining a current cache gain by updating the estimated reduction in the total execution cost for the dataflow of said first cache placement plan based on the updated number of references to said one or more executed operations;
• determining an alternative cache placement plan for the dataflow following said execution;
• and
• implementing said alternative cache placement plan when a difference between said current cache gain for said first cache placement plan and an estimated reduction in the total execution cost for the dataflow when said alternative cache placement plan is implemented satisfies a predefined threshold criteria.

An apparatus, comprising:

• a memory;
• and
• at least one processing device, coupled to the memory, operative to implement the following steps:
• obtaining a first cache placement plan for a dataflow comprised of a plurality of operations, wherein said first cache placement plan comprises a combination of output datasets of a subset of said operations in said dataflow to cache based on an estimated reduction in a total execution cost for the dataflow when said first cache placement plan is implemented given an input dataset;
• executing one or more operations of said dataflow and updating a number of references to said one or more executed operations to reflect remaining executions of said one or more executed operations in the dataflow;
• determining a current cache gain by updating the estimated reduction in the total execution cost for the dataflow of said first cache placement plan based on the updated number of references to said one or more executed operations;
• determining an alternative cache placement plan for the dataflow following said execution;
• and
• implementing said alternative cache placement plan when a difference between said current cache gain for said first cache placement plan and an estimated reduction in the total execution cost for the dataflow when said alternative cache placement plan is implemented satisfies a predefined threshold criteria.