Data shuffling with hierarchical tuple spaces

A system effective to shuffle source data to generate a dataset, the system comprising:

• a memory configured to store the source data;
• a processor configured to be in communication with the memory;
• at least a first map module and a second map module configured to be in communication with the memory and the processor;
• at least a shuffle module configured to be in communication with the memory and the processor;
• the first map module is configured to:
• generate first pair data from the source data;
• insert a first portion of the first pair data into a first local tuple space accessible by the first map module, wherein the inserted first portion of the first pair data includes keys that are absent from the first local tuple space prior to the insertion of the first portion of the first pair data;
• update the first local tuple space based on a second portion of the first pair data, wherein the second portion of the first pair data includes keys that are present in the first local tuple space prior to the update of the first local tuple space;
• the second map module is configured to:
• generate second pair data from the source data;
• insert a first portion of the second pair data into a second local tuple space accessible by the second map module, wherein the inserted first portion of the second pair data includes keys that are absent from the second local tuple space prior to the insertion of the first portion of the second pair data;
• update the second local tuple space based on a second portion of the second pair data, wherein the second portion of the second pair data includes keys that are present in the second local tuple space prior to the update of the second local tuple space;
• the processor is configured to:
• activate the shuffle module of the processor to execute a shuffle operation on pair data that includes a particular key;
• determine a presence of one or more keys in a global tuple space accessible by the first map module and the second map module;
• in response to a presence of an existing key in the global tuple space, update pair data in the global tuple space that includes the existing key, wherein the update is based on pair data among the first and second local tuple space that includes the existing key, and
• in response to an absence of a missing key in the global tuple space, insert pair data including the missing key among the first and second local tuple space into the global tuple space;
• the shuffle module is further configured to:
• identify a piece of pair data in the global tuple space that includes the particular key;
• fetch the identified pair data from a memory, wherein the fetched pair data includes the particular key and one or more values associated with the particular key;
• and
• execute the shuffle operation on the fetched pair data to generate the dataset.

A computer program product for shuffling source data to generate a dataset, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

• generate first pair data from the source data;
• generate second pair data from the source data, wherein each pair data among the first pair data and the second pair data includes a key and a value associated with the key;
• insert a first portion of the first pair data into a first local tuple space accessible by a first map module, wherein the inserted first portion of the first pair data includes keys that are absent from the first local tuple space prior to the insertion of the first portion of the first pair data;
• update the first local tuple space based on a second portion of the first pair data, wherein the second portion of the first pair data includes keys that are present in the first local tuple space prior to the update of the first local tuple space;
• insert a first portion of the second pair data into a second local tuple space accessible by a second map module, wherein the inserted first portion of the second pair data includes keys that are absent from the second local tuple space prior to the insertion of the first portion of the second pair data;
• update the second local tuple space based on a second portion of the second pair data, wherein the second portion of the second pair data includes keys that are present in the second local tuple space prior to the update of the second local tuple space;
• activate a shuffle phase indicated by the program instructions to execute a shuffle operation on pair data that includes a particular key;
• determine a presence of one or more keys in a global tuple space accessible by the first map module and the second map module;
• in response to a presence of an existing key in the global tuple space, update pair data in the global tuple space that includes the existing key, wherein the update is based on pair data among the first and second local tuple space that includes the existing key, and
• in response to an absence of a missing key in the global tuple space, insert pair data including the missing key among the first and second local tuple space into the global tuple space;
• identify pair data in the global tuple space that includes the particular key;
• fetch the identified pair data from a memory, wherein the fetched pair data includes the particular key and one or more values associated with the particular key;
• and
• execute the shuffle operation on the fetched pair data to generate the dataset.