A dynamic data processing pipeline

A system for dynamically generating a data processing pipeline, the system comprising:

• a memory;
• and
• a processor coupled to the memory, wherein the processor is configured to execute program instructions stored in the memory for:
• receiving data including one or more data types from a data source;
• creating a set of sub-pipelines based on the one or more data types, wherein each sub-pipeline of the set of sub-pipelines includes one or more processing layers;
• determining the one or more data types and volume of data assigned for processing to each processing layer of the one or more processing layers;
• and
• dynamically allocating resources to the one or more processing layers based on the one or more data types, the data source, and the volume of the data, wherein the dynamically allocating resources comprises determining a resource type, based on at least one of, the one or more data types, the data source, and the volume of data.

A computer programmable product having embodied thereon a computer program for dynamically generating a data processing pipeline, the computer programmable product storing instructions for:

• receiving data including one or more data types from a data source;
• creating a set of sub-pipelines based on the one or more data types, wherein each sub-pipeline of the set of sub-pipelines includes one or more processing layers;
• determining the one or more data types and volume of data assigned for processing to each processing layer of the one or more processing layers;
• and
• dynamically allocating resources to the one or more processing layers based on the one or more data types, the data source, and the volume of the data, wherein the dynamically allocating resources comprises determining a resource type, based on at least one of, the one or more data types, the data source, and the volume of data.