System and method for dynamically allocating computer resources to a data processing pipeline

A system for dynamically generating a data processing pipeline, the system comprising:

• a memory;
• and
• a processor coupled to the memory, wherein the processor is configured to execute program instructions stored in the memory for:
• receiving data including one or more data types from a data source;
• creating a set of sub-pipelines based on the one or more data types, wherein each sub-pipeline of the set of sub-pipelines includes one or more processing layers, wherein each processing layer handles different data processing and storage operations;
• determining the one or more data types and volume of data assigned for processing to each processing layer of the one or more processing layers, wherein the one more data types includes an application state data and a time interval-based application data;
• dynamically allocating resources to the one or more processing layers based on the data source, the volume of the data, the one or more data types, and a resource type which is one of a compute intensive resource and a memory intensive resource;
• and
• re-allocating the resources of the one or more processing layers to a subsequent processing layer based upon completion of the processing at the one or more processing layers.

A non-transitory computer program product having embodied thereon a computer program for dynamically generating a data processing pipeline, the computer program product storing instructions, which when executed by a processor, for:

• receiving data including one or more data types from a data source;
• creating a set of sub-pipelines based on the one or more data types, wherein each sub-pipeline of the set of sub-pipelines includes one or more processing layers, wherein each processing layer handles different data processing and storage operations;
• determining the one or more data types and volume of data assigned to each processing layer of the one or more processing layers, wherein the one more data types includes an application state data and a time interval-based application data;
• dynamically allocating resources to the one or more processing layers based on the data source, the volume of the data, the one or more data types, and a resource type which is one of a compute intensive resource and a memory intensive resource;
• and
• re-allocating the resources of the one or more processing layers to a subsequent processing layer based upon completion of the processing at the one or more processing layers.