DEVICE AND METHOD FOR OPTIMIZATION OF DATA PROCESSING IN A MapReduce FRAMEWORK
AssigneeThomson Licensing
InventorsNicolas Le Scouarnec, Erwan Le Merrer
Life of this patent
Estimating the term…
Abstract
A map reduce frame work for large scale data processing is optimized by the method of the invention that can be implemented by a master node. The method comprises reception of data from worker nodes on read pointer locations pointing to input data of tasks executed by these worker nodes and stealing of work from these tasks, the work being stolen being applied to input data that have not yet been processed by the task from which work is stolen.
Claims12
A method for processing data in a map reduce framework, wherein the method is executed by a master node and comprises:
- splitting of input data into input data segments;
- assigning tasks for processing said input data segments to worker nodes, where each worker node is assigned a task for processing an input data segment;
- determining, from data received from worker nodes executing said tasks, if a read pointer that points to a current read location in an input data segment processed by a task has not yet reached a predetermined threshold before input data segment end;
- and
- assigning of a new task to a free worker node, the new task being attributed a portion, referred to as split portion, of the input data segment that has not yet been processed by said task that has not yet reached a predetermined threshold before input data segment end, said split portion being a part of said input data segment that is located after said current read pointer location.
A master device for processing data in a map reduce framework, wherein said device comprises:
- a central processing unit for splitting of input data into input data segments;
- a central processing unit for assigning tasks for processing said input data segments to worker nodes, where each worker node is assigned a task for processing an input data segment;
- a central processing unit for determining, from data received from worker nodes executing said tasks, if a read pointer that points to a current read location in an input data segment processed by a task has not yet reached a predetermined threshold before input data segment end;
- and
- a central processing unit for assigning of a new task to a free worker node, the new task being attributed a portion, referred to as split portion, of the input data segment that has not yet been processed by said task that has not yet reached a predetermined threshold before input data segment end, said split portion being a part of said input data segment that is located after said current read pointer location.
Dependencies are inferred from the claim text.