# US 2021/0240519 A1 · Orchestration for data pipeline execution plans > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240519 A1 - Document ID: US20210240519A1 - Status: Published application - Kind: A1 - Application number: 16/779,040 - Filed: 2020-01-31 - Published: 2021-08-05 - Earliest claimed: 2020-01-31 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, Inc. - Inventors: Alex Gitelman; Twisha Mehta; Stephen Petschulat; Alex Field; LingTao Zhang - Web page: https://patentgrep.com/patent/US20210240519A1 ## CPC - G06F 9/485: {Task life-cycle, e.g. stopping, restarting, resuming execution (G06F9/4881 takes precedence)} - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)} - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)} - G06F 11/0772: {Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers} - G06F 11/3065: {Monitoring arrangements determined by the means or processing involved in reporting the monitored data (error or fault reporting or logging G06F11/0766)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and devices supporting dynamic process orchestration are described. An orchestration server may receive a request defining a data modification process from a user device. The orchestration server may generate an execution file based on the request, and the execution file may include a set of tasks for performing the data modification process and an order for performing the set of tasks. The orchestration server may execute, for the execution file, a first set of tasks according to the order for performing the set of tasks and, in some cases, may update the execution file based on executing the first subset of tasks. For example, updating the execution file may involve modifying a second subset of tasks of the set of tasks. The orchestration server may execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks. ## Claims ### Claim 1 A method for data processing, comprising: receiving, from a user device, a request defining a data modification process; generating an execution file based at least in part on the request, wherein the execution file comprises a set of tasks for performing the data modification process and an order for performing the set of tasks; executing, for the execution file, a first subset of tasks of the set of tasks according to the order for performing the set of tasks; updating the execution file based at least in part on executing the first subset of tasks, wherein the updating comprises modifying a second subset of tasks of the set of tasks; and executing, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks. ### Claim 2 (depends on 1) The method of claim 1, wherein executing the first subset of tasks and executing the modified second subset of tasks comprise: generating, on-the-fly for each task of the first subset of tasks and the modified second subset of tasks, a respective payload for each task indicating a data process and a system for performing the data process. ### Claim 3 (depends on 2) The method of claim 2, wherein the set of tasks comprises a first task and a second task, the method further comprising: executing, for the execution file, the first task according to the order for performing the set of tasks; and generating a payload for the second task based at least in part on an output of the executed first task. ### Claim 4 (depends on 2) The method of claim 2, wherein executing the first subset of tasks and executing the modified second subset of tasks comprise: communicating with a plurality of systems based at least in part on a plurality of generated payloads corresponding to the first subset of tasks and the modified second subset of tasks. ### Claim 5 (depends on 4) The method of claim 4, wherein communicating with the plurality of systems comprises: receiving, from a first system of the plurality of systems, a dataset as an output of executing a first task of the first subset of tasks or the modified second subset of tasks; storing, in intermediate storage, the dataset; and transmitting, to a second system of the plurality of systems, the dataset from the intermediate storage for executing a second task of the first subset of tasks or the modified second subset of tasks. ### Claim 6 (depends on 2) The method of claim 2, wherein the respective payload for each task comprises a respective JavaScript Object Notation file. ### Claim 7 (depends on 1) The method of claim 1, further comprising: receiving, from the user device, an abort command for the data modification process; and aborting execution of the execution file in response to the abort command, wherein a third subset of tasks of the set of tasks is unexecuted based at least in part on the aborted execution. ### Claim 8 (depends on 1) The method of claim 1, further comprising: sending, for display in a user interface of the user device, an indication of the data modification process and the set of tasks for performing the data modification process. ### Claim 9 (depends on 8) The method of claim 8, further comprising: receiving, from the user device, a user input to the user interface for a third subset of tasks of the set of tasks; further updating the execution file based at least in part on the user input, wherein the further updating comprises modifying the third subset of tasks; and executing, for the further updated execution file, the modified third subset of tasks according to the order for performing the set of tasks. ### Claim 10 (depends on 8) The method of claim 8, further comprising: identifying an error associated with executing a task of the first subset of tasks or the modified second subset of tasks; and sending, for display in the user interface of the user device, a notification indicating the error and the task of the first subset of tasks or the modified second subset of tasks. ### Claim 11 (depends on 8) The method of claim 8, further comprising: sending, for display in the user interface of the user device, a notification based at least in part on an output of executing the first subset of tasks, executing the modified second subset of tasks, or a combination thereof. ### Claim 12 (depends on 1) The method of claim 1, further comprising: hosting a plurality of data pipeline managers; and identifying that the request corresponds to a data pipeline manager of the plurality of data pipeline managers, wherein the execution file is generated by the data pipeline manager based at least in part on the request and processing logic for the data pipeline manager. ### Claim 13 (depends on 1) The method of claim 1, wherein the data modification process comprises one or more filter operations, one or more join operations, one or more grouping operations, one or more sorting operations, one or more format conversion operations, one or more validation operations, or a combination thereof. ### Claim 14 (depends on 1) The method of claim 1, wherein the request is received based at least in part on a user selection, a data processing schedule, an execution trigger, or a combination thereof. ### Claim 15 An apparatus for data processing, comprising: a processor; memory coupled with the processor; and instructions stored in the memory and executable by the processor to cause the apparatus to: receive, from a user device, a request defining a data modification process; generate an execution file based at least in part on the request, wherein the execution file comprises a set of tasks for performing the data modification process and an order for performing the set of tasks; execute, for the execution file, a first subset of tasks of the set of tasks according to the order for performing the set of tasks; update the execution file based at least in part on executing the first subset of tasks, wherein the updating comprises modifying a second subset of tasks of the set of tasks; and execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the instructions to execute the first subset of tasks and execute the modified second subset of tasks are further executable by the processor to cause the apparatus to: generate, on-the-fly for each task of the first subset of tasks and the modified second subset of tasks, a respective payload for each task indicating a data process and a system for performing the data process. ### Claim 17 (depends on 16) The apparatus of claim 16, wherein the set of tasks comprises a first task and a second task, and the instructions are further executable by the processor to cause the apparatus to: execute, for the execution file, the first task according to the order for performing the set of tasks; and generate a payload for the second task based at least in part on an output of the executed first task. ### Claim 18 (depends on 16) The apparatus of claim 16, wherein the instructions to execute the first subset of tasks and execute the modified second subset of tasks are further executable by the processor to cause the apparatus to: communicate with a plurality of systems based at least in part on a plurality of generated payloads corresponding to the first subset of tasks and the modified second subset of tasks. ### Claim 19 (depends on 18) The apparatus of claim 18, wherein the instructions to communicate with the plurality of systems are executable by the processor to cause the apparatus to: receive, from a first system of the plurality of systems, a dataset as an output of executing a first task of the first subset of tasks or the modified second subset of tasks; store, in intermediate storage, the dataset; and transmit, to a second system of the plurality of systems, the dataset from the intermediate storage for executing a second task of the first subset of tasks or the modified second subset of tasks. ### Claim 20 A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by a processor to: receive, from a user device, a request defining a data modification process; generate an execution file based at least in part on the request, wherein the execution file comprises a set of tasks for performing the data modification process and an order for performing the set of tasks; execute, for the execution file, a first subset of tasks of the set of tasks according to the order for performing the set of tasks; update the execution file based at least in part on executing the first subset of tasks, wherein the updating comprises modifying a second subset of tasks of the set of tasks; and execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks.