Interoperability between Map-Reduce and Distributed Array Runtimes
AssigneeMicrosoft Corporation
InventorsSudarshan Raghunathan, Damon Robert Hachmeister
Life of this patent
Estimating the term…
Loading patent…
AssigneeMicrosoft Corporation
InventorsSudarshan Raghunathan, Damon Robert Hachmeister
Estimating the term…
Described is a technology by which Map-Reduce runtimes and distributed array runtimes are interoperable. Map-Reduce chunks are processed into array data for processing in a distributed array runtime based upon merge information. A staging Map-Reduce job tags a chunk with tag information that indicates a relative position of the chunk in an array. A distributed array framework imports files produced via a Map-Reduce framework and provides an array to an application of the distributed array framework for processing. An export mechanism may output one or more Map-Reduce files from the distributed array framework.
In a computing environment, a method performed at least in part on at least one processor comprising, processing Map-Reduce chunks into array data for processing in a distributed array runtime, including accessing one or more files containing the chunks, in which the chunks are sorted by array position information, and assembling the chunks based upon merge information into the array data.
A system comprising, a distributed array framework configured to access files produced via a Map-Reduce framework, in which the files contain chunks of a distributed array sorted based upon array position information, and an import mechanism configured to convert data in the files containing the chunks into a data structure corresponding to an array containing array dimension information and array data, for processing of the array by an application of the distributed array framework.
One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising, executing a staging Map-Reduce job, including performing a staging mapping operation that tags a chunk with tag information that indicates a relative position of the chunk in an array.
Dependencies are inferred from the claim text.