Adapative system for processing distributed data files and a method thereof

A system for processing distributed data files, the system comprising:

• a plurality of machines, each comprising one or more processors;
• and
• a memory coupled with the one or more processors in each of the plurality of machines, wherein the one or more processors are configured to execute instructions stored in the memory to:
• receive one or more of a plurality of chunks of distributed data files from at least one data source and a set of instructions associated with the distributed data files, from a primary device, wherein the set of instructions comprises structured query transformers, and wherein the one or more of the distributed data files and the set of instructions are stored in the memory;
• generate processed data files by processing the one or more chunks of the distributed data files based on one or more characteristics of the distributed data files, wherein the one or more characteristics of the distributed data files comprise at least one of formats of the distributed data files, a schema of the distributed data files, one or more structured query transformers, and a processing requirement, and wherein the one or more chunks of the distributed data files are processed by:
• receiving an instance of the structured query transformers from one of the plurality of machines;
• and
• transforming the one or more chunks of the distributed data files by executing the instance of the structured query transformers on the one or more chunks of the distributed data files and while retaining an existing implementation of the system, wherein each of the plurality of machines is configured to process a different set of chunks from the plurality of chunks of the distributed data files using the set of instructions, and wherein the execution of the structured query transformers comprises steps of:
• generating a dependency graph based upon dependencies between the structured query transformers;
• and
• determining a sequence of execution of the structured query transformers based upon the dependency graph, wherein determining the sequence comprises reordering the sequence of execution of the structured query transformers based upon the dependencies;
• and
• transmit the processed data files to a data warehouse, thereby decoupling the processing of the distributed data files from the data warehouse.