US 2019/0197138 A1Published application

Data shuffling with hierarchical tuple spaces

AssigneeInternational Business Machines Corporation

InventorsAbdullah Kayi, Carlos Henrique Andrade Costa, Yoonho Park, Charles Johns

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods and systems for shuffling data to generate a dataset are described. A first map module may generate first pair data, and a second map module may generate second pair data, from source data. The first map module may insert the first pair data into a first local tuple space accessible to the first map module. The second map module may insert the second pair data into a second local tuple space accessible to the second map module. A shuffle module may request pair data that includes a particular key. The first and second pair data may be inserted into a global tuple space accessible by the first and second map modules. The shuffle module may identify the requested pair data in the global tuple space, and may fetch the identified pair data from a memory. The shuffle module may shuffle the fetched pair data to generate the dataset.

Claims20

Dependent claims

A method for shuffling source data to generate a dataset, the method comprising:

generating, by a first map module of a processor, first pair data from the source data;
generating, by a second map module of the processor, second pair data from the source data, wherein each pair data among the first pair data and the second pair data includes a key and a value associated with the key;
inserting, by the first map module of the processor, the first pair data into a first local tuple space accessible by the first map module;
inserting, by the second map module of the processor, the second pair data into a second local tuple space accessible by the second map module;
activating, by the processor, a shuffle module of the processor to execute a shuffle operation, on pair data that includes a particular key;
inserting, by the processor, and upon the activation of the shuffle module, the first pair data into a global tuple space accessible by the first map module and the second map module;
inserting, by the processor, and upon the activation of the shuffle module, the second pair data into the global tuple space;
identifying, by the shuffle module of the processor, pair data including the particular key in the global tuple space;
fetching, by the shuffle module of the processor, the identified pair data from a memory;
and
executing, by the shuffle module of the processor, a shuffle operation on the fetched pair data to generate the dataset, wherein the dataset includes the particular key and one or more values associated with the particular key.

8 more elements

A system effective to shuffle source data to generate a dataset, the system comprising:

a memory configured to store the source data;
a processor configured to be in communication with the memory;
at least a first map module and a second map module configured to be in communication with the memory and the processor;
at least one shuffle module configured to be in communication with the memory and the processor;
the first map module is configured to:
generate first pair data from the source data;
insert the first pair data into a first local tuple space accessible by the first map module;
the second map module is configured to:
generate second pair data from the source data;
insert the second pair data into a second local tuple space accessible by the second map module;
the processor is configured to:
activate the shuffle module to execute a shuffle operation on pair data that includes a particular key;
insert, upon the activation of the shuffle module, the first pair data into a global tuple space accessible by the first map module and the second map module;
insert, upon the activation of the shuffle module, the second pair data into the global tuple space;
the shuffle module is further configured to:
identify the pair data that includes the particular key in the global tuple space;
fetch the identified pair data from the memory;
and
execute the shuffle operation on the fetched pair data to generate the dataset, wherein the dataset includes the particular key and one or more values associated with the particular key.

16 more elements

A computer program product for shuffling source data to generate a dataset, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

generate first pair data from the source data;
generate second pair data from the source data, wherein each pair data among the first pair data and the second pair data includes a key and a value associated with the key;
insert the first pair data into a first local tuple space accessible by a first map module of the device;
insert the second pair data into a second local tuple space accessible by a second map module of the device;
activate a shuffle phase indicated by the program instructions to execute a shuffle operation on pair data that includes a particular key;
insert, upon the activation of the shuffle phase, the first pair data into a global tuple space accessible by the first map module and the second map module;
insert, upon the activation of the shuffle phase, the second pair data into the global tuple space;
identify pair data including the particular key in the global tuple space;
fetch the identified pair data from a memory;
and
execute the shuffle operation on the fetched pair data to generate the dataset, wherein the dataset includes the particular key and one or more values associated with the particular key.

8 more elements

Dependencies are inferred from the claim text.