Distributed method and apparatus for processing streaming data

A distributed apparatus for processing streaming data, comprising:

• at least one processor;
• and
• a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
• encapsulating received streaming data as a first resilient distributed dataset;
• performing a grouping operation on the first resilient distributed dataset based on time windows, the grouping operation comprising:
• assigning each data element in the first resilient distributed dataset into a group corresponding to a time window to which a recorded timestamp of the data element belongs, and forming second resilient distributed datasets comprising a limited number of data elements and respectively corresponding to the time windows;
• encapsulating the second resilient distributed datasets as a nested dataset comprising a plurality of the second resilient distributed datasets;
• and
• passing, using a predefined traversal operator, each of the second resilient distributed datasets in the nested dataset successively to a batch operator defined in a finite dataset to perform distributed data processing.

A non-transitory storage medium storing one or more programs, the one or more programs when executed by an apparatus, causing the apparatus to perform a distributed method for processing streaming data, the method comprising:

• encapsulating received streaming data as a first resilient distributed dataset;
• performing a grouping operation on the first resilient distributed dataset based on time windows, the grouping operation comprising:
• assigning each data element in the first resilient distributed dataset into a group corresponding to a time window to which a recorded timestamp of the data element belongs, and forming second resilient distributed datasets comprising a limited number of data elements and respectively corresponding to the time windows;
• encapsulating the second resilient distributed datasets as a nested dataset comprising a plurality of the second resilient distributed datasets;
• passing, using a predefined traversal operator, each of the second resilient distributed datasets in the nested dataset successively to a batch operator defined in a finite dataset to perform distributed data processing.