Distributed Data Scalable Adaptive Map-Reduce Framework

#Embodiments of the invention generally relate to information technology, and, more particularly, to a map reduce framework.

#Petascale analytics require an efficient map reduce framework that can leverage advances in processor technologies to improve the performance/cost ratio. Communication bottlenecks are a major roadblock for high throughput map-reduce over large quantities of data (for example, terabytes to petabytes). Currently, map-reduce frameworks suffer from such communication bottlenecks. Further, existing map-reduce frameworks are not able to leverage hybrid systems that provide accelerators. Additionally, the scheduling of the map reduce system is primarily centralized. Accordingly, there exists a need for distributed map reduce over large clusters.

#In one aspect of the present invention, a distributed data scalable adaptive map-reduce framework is provided. An exemplary computer-implemented method for generating a distributed data scalable adaptive map-reduce framework for at least one multi-core cluster can include steps of partitioning a cluster into at least one computational group, determining at least one key-group leader within each computational group, performing a local combine operation at each computational group, performing a global combine operation at each of the at least one key-group leader within each computational group based on a result from the local combine operation, and performing a global map-reduce operation across the at least one key-group leader within each computational group.

#FIG. 1 is an example cluster diagram illustrating an example embodiment, according to an aspect of the invention;

#FIG. 2 is a diagram illustrating an example embodiment, according to an aspect of the invention;

#FIG. 3 is a flow diagram illustrating techniques for generating a distributed data scalable adaptive map-reduce framework for at least one multi-core cluster, according to an embodiment of the invention; and

#As described herein, an aspect of the present invention includes an optimized distributed map reduce framework for large scale multi-core hybrid clusters. An embodiment of the invention includes a global combine phase and splitting batch mode map-reduce into multiple map-reduce iterations. A user can specify this combine function along with the final reduce function. Additionally, the framework detailed herein uses a two-level approach that enables a divide-conquer strategy for very large scale clusters and efficient communication within each computation group at a level.

#Accordingly, a data scalable designed framework as detailed herein provides significant performance and cost advantage over conventional map reduce approaches. For example, an aspect of the invention includes leveraging advances in process technologies to improve performance/cost ratio, as well as reducing communication bottlenecks due to high throughput map-reduce over large quantities of data (terabytes to petabytes).

#As described herein, an asynchronous map reduce algorithm includes mapping, combining locally, combining globally, and reducing globally. A cluster is partitioned into computational groups. Each computational group represents a locally dense group, which can include, for example, multiprocessor machines and tightly coupled clusters. Also, each map reduce computation is converted into an iterative map-combine then reduce operation. This can include a large number of smaller maps, combine-local, and/or combine-global operations, as well as one reduce operation per key.