Generating code for real-time stream processing
AssigneeInternational Business Machines Corporation
InventorsFrederic Chanson, Saniya Ben Hassen, Marc P. Yvon
Life of this patent
Estimating the term…
Abstract
A computer-implemented method for generating code for real-time stream processing, where data is streamed in tuples. One or more processors receive source code which includes code to be applied in a sequential series to one or more tuples. Processor(s) generate the code for real-time stream processing by: buffering received time sequence tuples in a buffer; and converting the code to be sequentially applied to the one or more tuples into code for buffered tuples with a loop construct. Processor(s) also generate code for: checking that all tuples in a processing window have been buffered; and applying the converted code to each tuple in the processing window.
Claims19
A computer-implemented method for generating an optimized code for processing a stream of tuples by balancing processing overhead and processing window size comprising:
- receiving, by one or more processors, tuple centric source code, wherein the tuple centric source code includes code to be applied in a sequential series to one tuple or a set of tuples at a time;
- receiving, by the one or more processors, a specified processing window size for a specified processing window based on a number of tuples to be received before processing the set of tuples;
- receiving, by the one or more processors, a specified dependence window size for a specified dependence window based on a number of earlier tuples that a current tuple depends on;
- and
- generating, by the one or more processors, code for real-time stream processing by:
- buffering the set of tuples in a buffer using the specified processing window;
- buffering all tuples that are required to process the current tuple using the specified dependence window;
- constructing a data structure containing a current window such that the data structure holds a total number of tuples in the specified processing window and the specified dependence window, wherein a runtime system checks consistency of the current window using the data structure;
- converting the tuple centric source code to be sequentially applied to the set of tuples into converted code for processing all buffered tuples with a loop construct, wherein processing starts with a first tuple for which all dependencies exist;
- generating code for checking that all tuples in the specified processing window have been buffered;
- and
- generating code for applying the converted code to each tuple in the specified processing window.
A computer system comprising one or more processors, one or more computer readable memories, and one or more computer readable storage mediums, and program instructions stored on at least one of the one or more computer readable storage mediums for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the stored program instructions comprising:
- program instructions to receive tuple centric source code, wherein the tuple centric source code includes code to be applied in a sequential series to one tuple or a set of tuples at a time;
- program instructions to receive a specified processing window size for a specified processing window based on a number of tuples to be received before processing the set of tuples;
- program instructions to receive a specified dependence window size for a specified dependence window based on a number of earlier tuples that a current tuple depends on;
- and
- program instructions to generate code for real-time stream processing by:
- buffering the set of tuples in a buffer using the specified processing window;
- buffering all tuples that are required to process the current tuple using the specified dependence window;
- constructing a data structure containing a current window such that the data structure holds a total number of tuples in the specified processing window and the specified dependence window, wherein a runtime system checks consistency of the current window using the data structure;
- converting the tuple centric source code to be sequentially applied to the set of tuples into converted code for processing all buffered tuples with a loop construct, wherein processing starts with a first tuple for which all dependencies exist;
- generating code for checking that all tuples in the specified processing window have been buffered;
- and
- generating code for applying the converted code to each tuple in the specified processing window.
A computer program product for generating an optimized code for processing a stream of tuples by balancing processing overhead and processing window size, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
- receive tuple centric source code, wherein the tuple centric source code includes code to be applied in a sequential series to one tuple or a set of tuples at a time;
- receive a specified processing window size for a specified processing window based on a number of tuples to be received before processing the set of tuples;
- receive a specified dependence window size for a specified dependence window based on a number of earlier tuples that a current tuple depends on;
- and
- generate code for real-time stream processing by:
- buffering the set of tuples in a buffer using the specified processing window;
- buffering all tuples that are required to process the current tuple using the specified dependence window;
- constructing a data structure containing a current window such that the data structure holds a total number of tuples in the specified processing window and the specified dependence window, wherein a runtime system checks consistency of the current window using the data structure;
- converting the tuple centric source code to be sequentially applied to the set of tuples into converted code for processing all buffered tuples with a loop construct, wherein processing starts with a first tuple for which all dependencies exist:
- generating code for checking that all tuples in the specified processing window have been buffered;
- and
- generating code for applying the converted code to each tuple in the specified processing window.
Dependencies are inferred from the claim text.