Task orchestration method for data processing, orchestrator, device and readable storage medium

An orchestrator for data processing, comprising:

• a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program, when being executed by the processor, causes the processor to:
• acquire task scripts, wherein the task scripts are descriptive language scripts;
• automatically generate data dependency relationships between source tables and target tables in tasks corresponding to the task scripts according to the task scripts;
• generate a data-driven trigger chain of the tasks according to the data dependency relationships between the source tables and the target tables of the tasks, wherein a beginning of the data-driven trigger chain is original data;
• drive execution of the tasks according to the data-driven trigger chain, wherein the computer program, when being executed by the processor, further causes the processor to:
• extract, from syntax trees of the task scripts, data partitions of the source tables that serve as input data of corresponding tasks, and data partitions of the target tables that serve as output data of the corresponding tasks, wherein the data partitions of the target tables depend on the data partitions of the source tables;
• and
• generate the data dependency relationships between the data partitions of the source tables and the data partitions of the target tables based on a middle variable according to the syntax trees.

A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor of an orchestrator for data processing, causes the orchestrator to:

• acquire task scripts, wherein the task scripts are descriptive language scripts;
• automatically generate data dependency relationships between source tables and target tables in tasks corresponding to the task scripts according to the task scripts;
• generate a data-driven trigger chain of the tasks according to the data dependency relationships between the source tables and the target tables of the tasks, wherein a beginning of the data-driven trigger chain is original data;
• drive execution of the tasks according to the data-driven trigger chain, wherein the computer program, when being executed by the processor, further causes the processor to:
• extract, from syntax trees of the task scripts, data partitions of the source tables that serve as input data of corresponding tasks, and data partitions of the target tables that serve as output data of the corresponding tasks, wherein the data partitions of the target tables depend on the data partitions of the source tables;
• and
• generate the data dependency relationships between the data partitions of the source tables and the data partitions of the target tables based on a middle variable according to the syntax trees.