Declarative specification of data integration workflows for execution on parallel processing platforms

A method comprising:

• receiving a declarative specification that includes a plurality of stages, each stage specifying an atomic operation, a data input to the atomic operation, and a data output from the atomic operation, the data input characterized by a data type, the data type being at least one of structured data, semi-structured data, and unstructured data;
• reducing a complexity of the declarative specification by generating links between at least two of the plurality of stages to create a data integration workflow, the at least two of the plurality of stages in the data integration workflow having data inputs characterized by different data types;
• compiling, on a computer, the data integration workflow to generate computer code for execution on a parallel processing platform, the computer code including instructions configured for:
• accessing subsets of interrelated data corresponding to the data inputs,
• applying a mapping operation in parallel to the subsets of the interrelated data to generate a plurality of output groups, and
• merging the output groups according to the data integration workflow to generate at least one output of the data outputs, the at least one output being characterized by a singular data type;
• optimize with an optimum number of mappers and reducers the computer code for execution on the parallel processing platform;
• performing a filter operation via the computer code to retrieve a phone number, the filter operation including:
• accessing structured data subset corresponding to a first data input, the structured data subset including a name,
• retrieving the phone number from a look-up table with respect to the name,
• retrieving the phone number from unstructured data subset corresponding the first data input by utilizing a rule based annotator to extract numbers from natural language of the unstructured data subset when the phone number is not initially retrieved from the look-up table,
• extracting positive or negative sentiments from unstructured data of a message field of the at least one email corresponding to the phone number, and
• outputting the at least one email as the at least one output when a negative sentiment is extracted from the unstructured data of the message field;
• and
• update a sentiment field in accordance with the positive or negative sentiments extracted from the unstructured data of the message field,
• wherein the computer code is a MapReduce application.