Method and apparatus for data integration framework

A system for integrating data, comprising:

• a memory and a processor operatively connected to the memory via a communication network,
• wherein the processor is configured to:
• implement a data processing framework configured to run native on a big data platform;
• abstract data processing constructs to a user friendly template, thereby eliminating necessity of user initiated tasks of instantiating language level objects;
• implement a core set of data pipeline configurations on the template configured to initiate a chain of user defined data transformations;
• receive input of the chain of the user defined data transformations;
• test each transformation independently of each other;
• and
• output data integration solutions on the big data platform based on a positive test result.

A non-transitory computer readable medium configured to store instructions for integrating data, wherein when executed, the instructions cause a processor to perform the following:

• implementing a data processing framework configured to run native on a big data platform;
• abstracting data processing constructs to a user friendly template, thereby eliminating necessity of user initiated tasks of instantiating language level objects;
• implementing a core set of data pipeline configurations on the template configured to initiate a chain of user defined data transformations;
• receiving input of the chain of the user defined data transformations;
• testing each transformation independently of each other;
• and
• outputting data integration solutions on the big data platform based on a positive test result.