Stream processing without central transportation planning

A computer program product for stream processing using a virtual extensible local area network (VXLAN), the computer program product comprising a computer readable storage medium having program instructions embodied therewith to:

• receive an operator graph of a streaming application, the operator graph comprising a plurality of processing elements, wherein each processing element comprises one or more operators of a plurality of operators;
• use a group of VXLAN segments between the plurality of processing elements without transmission control protocol (TCP) connections, wherein a VXLAN segment of the group of VXLAN segments is used between each pair of processing elements of the plurality of processing elements to exchange data based on the operator graph;
• receive input data by a first processing element of the plurality of processing elements;
• process the input data by one or more operators of the streaming application in the first processing element;
• multicast output data, by the first processing element, to each of the VXLAN segments of the group of VXLAN segments, wherein each of the other processing elements of the plurality of processing elements receive the output data;
• receive the output data from the first processing element by a second processing element on a given VXLAN segment of the group of VXLAN segments;
• in response to receiving the output data from the first processing element, determine, by the second processing element, whether the second processing element is an intended recipient of the output data;
• if the second processing element is an intended recipient of the output data, process the output data by one or more operators of the streaming application in the second processing element;
• and
• if the second processing element is not the intended recipient of the output data, ignore the output data by the second processing element.

A system comprising:

• a processor;
• and
• a computer readable storage medium having program instructions embodied therewith for stream processing using a virtual extensible local area network (VXLAN), wherein the program instructions are executable to:
• receive an operator graph of a streaming application, the operator graph comprising a plurality of processing elements, wherein each processing element comprises one or more operators of a plurality of operators;
• use a group of VXLAN segments between the plurality of processing elements without transmission control protocol (TCP) connections, wherein a VXLAN segment of the group of VXLAN segments is used between each pair of processing elements of the plurality of processing elements to exchange data based on the operator graph;
• receive input data by a first processing element of the plurality of processing elements;
• process the input data by one or more operators of the streaming application in the first processing element;
• multicast output data, by the first processing element, to each of the VXLAN segments of the group of VXLAN segments, wherein each of the other processing elements of the plurality of processing elements receive the output data;
• receive the output data from the first processing element by a second processing element on a given VXLAN segment of the group of VXLAN segments;
• in response to receiving the output data from the first processing element, determine, by the second processing element, whether the second processing element is an intended recipient of the output data;
• if the second processing element is an intended recipient of the output data, process the output data by one or more operators of the streaming application in the second processing element;
• and
• if the second processing element is not the intended recipient of the output data, ignore the output data by the second processing element.