Distributed method and apparatus for processing streaming data

A distributed apparatus for processing streaming data, comprising:

• a first encapsulating unit for encapsulating received streaming data as a first resilient distributed dataset;
• a grouping unit for performing a grouping operation of the first resilient distributed dataset based on time windows, the grouping operation comprising:
• assigning each data element in the first resilient distributed dataset into a group corresponding to the time window to which a recorded timestamp of the data element belongs, and forming second resilient distributed datasets, each of the second resilient distributed datasets comprising a limited number of data elements and corresponding to the time window;
• a second encapsulating unit for encapsulating the second resilient distributed datasets as a nested dataset comprising a plurality of the second resilient distributed datasets;
• and
• an execution unit for passing, using a predefined traversal operator, each of the second resilient distributed datasets in the nested dataset successively to a batch operator defined in a finite dataset to perform distributed data processing.

A non-transitory storage medium storing one or more programs, the one or more programs when executed by an apparatus, causing the apparatus to perform a distributed method for processing streaming data, the method comprising:

• encapsulating received streaming data as a first resilient distributed dataset;
• performing a grouping operation of the first resilient distributed dataset based on time windows, the grouping operation comprising:
• assigning each data element in the first resilient distributed dataset into a group corresponding to the time window to which a recorded timestamp of the data element belongs, and forming second resilient distributed datasets, each of the second resilient distributed datasets comprising a limited number of data elements and corresponding to the time window;
• encapsulating the second resilient distributed datasets as a nested dataset comprising a plurality of the second resilient distributed datasets;
• passing, using a predefined traversal operator, each of the second resilient distributed datasets in the nested dataset successively to a batch operator defined in a finite dataset to perform distributed data processing.