System and method for linearizing messages from data sources for optimized high-performance processing in a stream processing system

A computer program product comprising a non-transient, machine-readable medium storing instructions which, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

• receiving a data object at a distributed process from a data source, the data object including a plurality of files;
• passing the plurality of files to the file staging area of the distributed process, the passing of the plurality of files at the file staging area is based on determining that a task of the distributed process is unable to operate on the data object;
• generating a tuple from the data object received from the data source, the tuple comprising a first field and a second field, the first field containing a file reference referring to the plurality of files in the file staging area, and the second field being configured to store a value indicative that the first field contains the file reference;
• unbundling the plurality of files at the file staging area by an unbundling task of the distributed process, the unbundling task configured to create at least one batch file in the file staging area based on at least the value in the second field stored at the tuple;
• extracting a portion of files associated with the file reference from the plurality of files to generate a processed tuple;
• converting the portion of files in the processed tuple into an output tuple based at least on a native string format of the file reference associated with the portion of files;
• and
• transmitting the output tuple to a data sink.

A system comprising:

• at least one programmable processor to perform operations comprising:
• receiving a data object at a distributed process from a data source, the data object including a plurality of files;
• passing the plurality of files to a file staging area of the distributed process, the passing of the plurality of files at the file staging area being based on determining that a task of the distributed process is unable to operate on the data object;
• generating a tuple from the data object received from the data source, the tuple comprising a first field and a second field, the first field containing a file reference referring to the plurality of files in the file staging area, and the second field being configured to store a value indicative that the first field contains the file reference;
• unbundling the plurality of files at the file staging area by an unbundling task of the distributed process, the unbundling task configured to create at least one batch file in the file staging area based on at least the value in the second field stored at the tuple;
• extracting a portion of files associated with the file reference from the plurality of files to generate a processed tuple;
• converting the portion of files in the processed tuple into an output tuple based at least on a native string format of the file reference associated with the portion of files;
• and
• transmitting the output tuple to a data sink.