US 9,313,133 B2 · Anticipatory warm-up of cluster resources for jobs processed on multiple cluster nodes

Abstract

Systems and methods are disclosed for reducing latency in processing data sets in a distributed fashion. A job-queue operable for queuing data-processing jobs run on multiple nodes in a cluster may be communicatively coupled to a job analyzer. The job analyzer may be operable to read the data-processing jobs and extract information characterizing those jobs in ways that facilitate identification of resources in the cluster serviceable to run the data-processing jobs and/or data to be processed during the running of those jobs. The job analyzer may also be coupled to a resource warmer operable to warm-up a portion of the cluster to be used to run a particular data-processing job prior to the running of the job. In some embodiments, mappers and/or reducers may be extracted from the jobs and converted into compute node identifiers and/or data units identifying blocks for processing, informing the warm-up operations of the resource warmer.

Claims20

Dependent claims

A system for processing large data sets comprising:

a job queue operable for queuing multiple data-processing jobs, each data-processing job comprising multiple component tasks, to be run in a distributed fashion on multiple nodes, each node comprising at least one of physical memory and at least one processor, in a cluster of nodes;
and
a job analyzer communicatively coupled to the job queue and operable to:
read the multiple data-processing jobs in the job queue, and
extract characterization information from the multiple data-processing jobs that characterizes:
at least one resource in the cluster serviceable to run the multiple data-processing jobs, and
data to be processed during the multiple data-processing jobs;
and
an advance module, coupled to the job analyzer, operable to initiate processing, contrary to a default approach to processing data-processing jobs in the job queue, of a set of data-processing jobs, upon warm-up operations satisfying a threshold for loading data blocks to be processed by the set into cache for mapper in a cluster portion identified by characterization information for the set, in anticipation of processing the set.

6 more elements

A method for processing data sets comprising:

reading, in advance of processing a set of data-processing jobs in a job queue, multiple data-processing jobs, comprising the set of data-processing jobs, in the job queue, each data-processing job in the multiple data-processing jobs configured for distributed processing of a data units on a cluster of nodes, the data blocks stored at multiple locations;
extracting, in advance of processing the set of data-processing jobs in the job queue, characterization information from the multiple data-processing jobs, the characterization information having information characterizing resources in the cluster for processing the multiple data-processing jobs;
and
initiating processing of, contrary to a default approach to processing data-processing jobs in the job queue, the set of data-processing jobs, upon warm-up operations satisfying a threshold for loading data blocks to be processed by the set into cache for mapper in a cluster portion identified by characterization information for the set, in anticipation of running the set.

1 more element

A data processing system comprising:

a cluster of nodes, each node having at least one of physical memory, a processor, and software for coordinating at least one of mapping and reducing operations within the cluster;
a distribution module operable to:
divide an input data file into multiple splits,
store at least one copy of each of the multiple splits as data units at multiple locations selected from at least one of the cluster of nodes and a backend storage device;
a name node within the cluster of nodes receiving locational awareness identifying the multiple locations of the data blocks;
a job analyzer operable to retrieve characterization information comprising a set of mappers for each of multiple a-data-processing jobs in a job queue;
and
an advance module, coupled to the job analyzer, operable to initiate processing of, contrary to a default approach to processing data-processing jobs in the job queue, a set of data-processing jobs, upon warm-up operations satisfying a threshold for loading data blocks to be processed by the set into cache for mapper in a cluster portion identified by characterization information for the set, in anticipation of running the test.

5 more elements

Dependencies are inferred from the claim text.

Anticipatory warm-up of cluster resources for jobs processed on multiple cluster nodes

Life of this patent

Anticipatory warm-up of cluster resources for jobs processed on multiple cluster nodes

Life of this patent

Abstract

Claims20