# US 2013/0297624 A1 · Interoperability between Map-Reduce and Distributed Array Runtimes

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0297624 A1
- Document ID: US20130297624A1
- Status: Published application
- Kind: A1
- Application number: 13/465,394
- Filed: 2012-05-07
- Published: 2013-11-07
- Earliest claimed: 2012-05-07
- Assignees: Microsoft Corporation
- Inventors: Sudarshan Raghunathan; Damon Robert Hachmeister
- Web page: https://patentgrep.com/patent/US20130297624A1

## CPC

- G06F 16/334: Query execution (filtering based on additional data G06F16/335)
- G06F 16/10: File systems; File servers
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described is a technology by which Map-Reduce runtimes and distributed array runtimes are interoperable. Map-Reduce chunks are processed into array data for processing in a distributed array runtime based upon merge information. A staging Map-Reduce job tags a chunk with tag information that indicates a relative position of the chunk in an array. A distributed array framework imports files produced via a Map-Reduce framework and provides an array to an application of the distributed array framework for processing. An export mechanism may output one or more Map-Reduce files from the distributed array framework.

## Claims

### Claim 1

In a computing environment, a method performed at least in part on at least one processor comprising, processing Map-Reduce chunks into array data for processing in a distributed array runtime, including accessing one or more files containing the chunks, in which the chunks are sorted by array position information, and assembling the chunks based upon merge information into the array data.

### Claim 2 (depends on 1)

The method of claim 1 further comprising, processing the chunks in a staging Map-Reduce job to sort the chunks, including tagging the chunks with relative array position information, partitioning the chunks based upon a number of reducers, sorting the chunks based upon the relative array position information into sorted chunks, and providing the chunks to the reducers based upon the sorting

### Claim 3 (depends on 2)

The method of claim 2 further comprising, at each reducer, writing a file to a distributed file system.

### Claim 4 (depends on 1)

The method of claim 1 further comprising, exporting results of the processing in a distributed array runtime into one or more Map-Reduce output data files.

### Claim 5 (depends on 4)

The method of claim 4 wherein exporting the results comprises writing the one or more Map-Reduce output data files to a distributed file system.

### Claim 6 (depends on 1)

The method of claim 1 wherein the chunks correspond to row or columns vectors, and further comprising, sorting the chunks based upon row position information or column position information.

### Claim 7 (depends on 1)

The method of claim 1 wherein the chunks correspond to hyperplanes, and further comprising, sorting the chunks based upon hyperplane position information.

### Claim 8 (depends on 1)

The method of claim 1 wherein accessing the one or more files comprises reading the files from a distributed file system.

### Claim 9 (depends on 1)

The method of claim 1 further comprising, obtaining the merge information from metadata included in the one or more files.

### Claim 10

A system comprising, a distributed array framework configured to access files produced via a Map-Reduce framework, in which the files contain chunks of a distributed array sorted based upon array position information, and an import mechanism configured to convert data in the files containing the chunks into a data structure corresponding to an array containing array dimension information and array data, for processing of the array by an application of the distributed array framework.

### Claim 11 (depends on 10)

The system of claim 10 wherein the import mechanism converts the data into the data structure based upon merge information.

### Claim 12 (depends on 10)

The system of claim 10 further comprising a staging mechanism of the map reduce framework, in which the staging mechanism includes one or more mappers that each tags chunks with relative array position information, and a sort mechanism configured to produce one or more of the files sorted based upon the relative position information.

### Claim 13 (depends on 12)

The system of claim 12 wherein the staging mechanism further includes a partitioner configured to associate tagged chunks with reducers, in which the sort mechanism determines arranges the chunks for the reducers based upon the relative array position information.

### Claim 14 (depends on 10)

The system of claim 10 wherein the distributed array framework includes a distributed array framework library.

### Claim 15 (depends on 10)

The system of claim 10 wherein the Map-Reduce framework comprises a Hadoop™-based runtime environment in which the files are accessed via a distributed file system.

### Claim 16 (depends on 10)

The system of claim 10 further comprising an export mechanism configured to output one or more Map-Reduce files from the distributed array framework.

### Claim 17 (depends on 10)

The system of claim 10 wherein the array comprises a multidimensional numeric array.

### Claim 18

One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising, executing a staging Map-Reduce job, including performing a staging mapping operation that tags a chunk with tag information that indicates a relative position of the chunk in an array.

### Claim 19 (depends on 18)

The one or more computer-readable media of claim 18 having further computer-executable instructions comprising, sorting a plurality of chunks based upon associated tag information for output as a sorted set of file data in which the ordering of the sorted set of file data is based upon the associated tag information.

### Claim 20 (depends on 18)

The one or more computer-readable media of claim 18 having further computer-executable instructions comprising, processing the sorted set of file data into array data, in which the position of each chunk in the array data is based upon the ordering of the sorted set of file data.
