Dynamically generating an optimized processing pipeline for tasks
AssigneeInternational Business Machines Corporation
InventorsLuke Taher, Diogo Alexandre Ferreira Ramos, Vinh Tuan Thai
Life of this patent
Estimating the term…
Abstract
A method for dynamically generating an optimized processing pipeline for tasks is provided. The method identifies one or more tasks to be executed from defined tasks that are defined declaratively as a number of stages of input data, data transformations, and output data. The method processes the identified tasks to determine dependencies between the tasks based on their defined stages and creates one or more optimized data processing pipelines by performing a dependency resolution procedure on stages of all tasks in parallel using the task dependencies to determine the order of the stages and removing duplication of stages between tasks.
Claims20
A computer-implemented method for dynamically generating an optimized processing pipeline for tasks, comprising:
- identifying tasks to be executed from defined tasks that are defined declaratively as a number of stages of input data, data transformations, and output data;
- processing the identified tasks to determine dependencies between the tasks based on defined stages of tasks;
- and
- creating one or more optimized data processing pipelines by performing a dependency resolution procedure on stages of all tasks in parallel using the determined dependencies to determine the order of stages and removing duplication of stages between the tasks.
A system for dynamically generating an optimized processing pipeline for tasks, comprising:
- a processor and a memory configured to provide computer program instructions to the processor to execute functions of a plurality of components:
- a task identifying component for identifying tasks to be executed from defined tasks that are defined declaratively as a number of stages of input data, data transformations, and output data;
- a dependencies component for processing the identified tasks to determine dependencies between the tasks based on defined stages of the tasks;
- and
- a pipeline creating component for creating one or more optimized data processing pipelines by performing a dependency resolution procedure on stages of all tasks in parallel using the determined dependencies to determine the order of stages and removing duplication of stages between the tasks.
A computer program product for dynamically generating an optimized processing pipeline for tasks, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
- identifying tasks to be executed from defined tasks that are defined declaratively as a number of stages of input data, data transformations, and output data;
- processing the identified tasks to determine dependencies between the tasks based on defined stages of tasks;
- and
- creating one or more optimized data processing pipelines by performing a dependency resolution procedure on stages of all tasks in parallel using the determined dependencies to determine the order of stages and removing duplication of stages between the tasks.
Dependencies are inferred from the claim text.