Enhanced Handling Of Intermediate Data Generated During Distributed, Parallel Processing

Abstract

Systems and methods are disclosed for reducing latency in shuffle-phase operations employed during the MapReduce processing of data. One or more computing nodes in a cluster of computing nodes capable of implementing MapReduce processing may utilize memory servicing such node(s) to maintain a temporary file system. The temporary file system may provide file-system services for intermediate data generated by applying one or more map functions to the underlying input data to which the MapReduce processing is applied. Metadata devoted to this intermediated data may be provided to and/or maintained by the temporary file system. One or more shuffle operations may be facilitated by accessing file-system information in the temporary file system. In some examples, the intermediate data may be transferred from one or more buffers receiving the results of the map function(s) to a cache apportioned in the memory to avoid persistent storage of the intermediate data.

Dependencies are inferred from the claim text.