Workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration
AssigneeSamsung Electronics Co., Ltd.
InventorsMyung-June Jung, Ju-Pyung Lee
Life of this patent
Estimating the term…
Abstract
A workload-aware distributed data processing apparatus and method for processing large data based on hardware acceleration are provided. The data processing apparatus includes a memory buffer including partitions. The data processing apparatus further includes a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme. The data processing apparatus further includes a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result.
Claims18
A data processing apparatus comprising:
- a memory buffer comprising partitions;
- a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme;
- and
- a reduce node configured to receive content of a corresponding one of the partitions, and
- perform a reduction operation on the content to generate a reduce result,
- wherein:
- the mapping result comprises a key;
- and
- the partition unit comprises a main processor configured to generate an intermediate code with a fixed size based on the key, and
- a proportion controller configured to determine the partitions based on the intermediate code and the partition proportion scheme, generate partition codes
- corresponding to the respective partitions.
A data processing apparatus comprising:
- a memory buffer comprising partitions;
- a partition unit configured to distribute a mapping result to the partitions based on a partition proportion scheme;
- and
- a reduce node configured to receive content of a corresponding one of the partitions, and perform a reduction operation on the content to generate a reduce result;
- and
- a mapping node configured to process input data in parallel to generate the mapping result,
- wherein:
- the mapping result comprises a key;
- and
- the mapping node is further configured to
- discard a bit other than a least significant bit of the key to update the key if a length of the key is greater than the least significant bit, and
- add a padding bit to the least significant bit to update the key if the length of the key is less than the least significant bit.
A data processing method comprising:
- distributing a mapping result to partitions of a memory buffer based on a partition proportion scheme;
- determining content of a corresponding one of the partitions;
- and
- performing a reduction operation on the content to generate a reduce result,
- wherein:
- the mapping result comprises a key;
- and
- the distributing of the mapping result comprises generating an intermediate code with a fixed size based on the key,
- determining the partitions based on the intermediate code and the partition proportion scheme, and
- generating partition codes corresponding to the respective partitions.
Dependencies are inferred from the claim text.